Many people blindly trust free online lens test sites such as DxO, OpticalLimits , Lenstip, etc. But their testing is not really complete, and sometimes even misguiding.
The problem of the online based lens tests is that they all use either Immatest or DxO for all their so called scientific lens tests(nothing scientific about it though).
Many times ,Cameralabs and Fred Miranda tests have clearly illustrated how useless or even plain wrong the Immatest based resolution chart test could go, but there are many people still argue against Fred's excellent real practical lens tests as it is too subjective.
Sure his tests are a bit subjective, focusing a bit too much on rendering, but still it is the only kind of lens test online to show us how these lenses he tests actually perform in real life scene.
Immatest based tests can be really good and useful, but it alone does not show any thing more than how the lenses tested perform in ten feet focus range. So I think Cameralabs lens test are arguably the most complete one right now we can read online, as Gordon tests every lens both at infinity and Immatest focus distances.
Why most of Imatest or DxO based lens tests are sometimes very inaccurate , or simply misleading, and I think that's the reason why the Batis lenses are so much underrated(updated 7):
First of all, many people seem wondering which online lens tests are useful or good or simply the best....
The DxO or Imatest chart based simple so-called lens resolution tests are really too simple , it cannot measure resolution outside of the flat chart plane, or high frequency details, or minor contrast contrast level, etc. And all those test chart based lens tests simply measure lenses for ten feet focus distance resolution, so it is quite useless for calculating or speculating actual resolution of any lens for macro distance or infinity or portrait distance say 5 feet focus distane, etc.
It is simply too incomplete......as a lens test....tells us almost nothing about real life performance of any lens they test....
So the answer, for me, is “almost none of them” is useful....Maybe except a very few such as Fred Miranda, Roger Cicala, and Lenstip as they tests much more than just resolution on a flat panel plane.
Sure, that on the surface appears controversial and goes against the grain of thought in a world where everyone wants ease and simplicity; your food delivered in an hour, ATM cash at your fingertips, and single number lens scores or grades. For landscape work, and even for a lot of other works like portrait, architecture,etc., things just aren’t that simple. Sorry there is no free lunch. If you really need a great lens test you might have to do it yourself or just need to subscribe a subscription based expensive site.
Anyway. to understand why I say this, we first have to take a look at the challenges faced by a lens designer. Ultimately, all lenses are trade-offs in some aspect or another. It might be a cost constraint, it might be a design decision borne from the ability of manufacturing to reliably and consistently manufacture many of said lens cases, it might be simply physics, it might be a balancing act that incorporates what the designer values for the lens, what *task* the lens is meant to do. And these trade-offs might very well mean that a lens that would be brilliant for a photographer in the real world won’t test well on the typical “test site” lens tests. But we must start with that – all lenses are trade-offs, and as such, no lens, no matter what brand, is perfect, even a Leica or a Rodenstock.
The second part of the discussion is to look at how lenses can be evaluated objectively. The gold standard for lens resolution evaluation is the optical MTF bench. Results from these are not what you see when you go to an old photozone (now opticallimits) chart, or a DXO lens or resolution score, or the imatest graphs from many sites. The only way you’re likely to see optical bench MTF is when Roger Cicala from lens rentals. com runs a wide open test on his MTF bench across a set of samples, and those can be found in that sites blog, and cutting to the chase first, I’ll take that information, as limited as it is relative to what the ideal would be (MTF stopped down as well) all day over the other sites. I’ll explain why later. You can of course find computer (theoretical) MTF on many of the manufacturers web sites; this comes from the design software – a real lens will never be this good. Zeiss provides measured (not theoretical) MTF and also includes stopped down, so their information is far more useful than anyone else’s. Optical bench MTF tells us resolution at a particular contrast at a particular part along the axis of the frame. Resolution and contrast are always intertwined – one can’t exist without the other. The typical form of MTF we see is multiple traces which represent different frequencies against % contrast, at various spots along the frame. Always wide open except for Zeiss or the rare time Roger runs one stopped down. MTF from the lens designers “view” might come in an orientation where it’s plotted as contrast versus frequency, with multiple traces showing different points along the frame.
This is actually the key to it all, even though we never see it. Years ago, one of the first pioneers of evaluation of image quality using MTF, a gentleman from RCA named Otto Schade, determined that image quality of a lens could best be determined by looking at the *area* under the trace when viewed in resolution plotted against contrast. This has huge implications as we’ll see in a bit. It basically tells us that looking at contrast at one resolution is only a small glimpse of what the lenses performance is – like judging the predominance of red cars in a 5000 car lot by looking at a quadrant of 4 cars.
The problem is that optical bench MTF is expensive and requires a trained tech to run. It’s well out of the range of your typical web site. As such, an entire industry (almost) was created where slant edge test charts are analyzed and a number at a certain contrast point is calculated from the chart photographed by the camera with the lens on it. This presents several problems:
First, the chart has to be in close/moderate distance. You aren’t putting charts on trees across the lake. Thing is, within the context of trade-offs of lens design, performance in closer ranges doesn’t always equal performance at distance.
Secondly, the fact that we’re using a single spot (the 4 car quadrant instead of the entire parking lot) means we’re not actually anywhere near “the absolute, perfect answer” with regards to the image quality of the lens.
But chart based testing is relatively easy and affordable, and while it has (to me, and others) significant flaws, still offers some value – as long as we understand the limitations, which is the point, really, of my long reply to you.
An example: sorry for Canon and Sony guys, but I think the AF-S58mm f1.4G might be the best example case here, so let’s take the 58/1.4G, a portrait specific lens designed with definite trade-off navigation to sacrifice close range resolution to improve bokeh and focus transition – the interviews with the designer himself state this – it’s not a guess. How do you think this lens measures on a test chart? Quite poorly. Often the laughing stock, because in the test chart range, the designer figured a photographer doesn’t *require* the higher frequency detail and would benefit from the bokeh and OOF transitions in real life shooting. So the lens scores poorly on all the test sites – yet when we look at the wide open MTF that Roger ran a while back, we see it’s actually quite good – not world class, but quite good, better than many garden variety 50mm class lenses. Why? Because the test charts distance sits squarely in a point where the designer made a horse trade, while the optical bench MTF, which looks at infinity performance, does not. See how a lens that would test poorly on almost all test chart sites could actually be a pretty decent one for landscape , portrait, architecture and other apps now?
In the end, I always evaluate lenses I consider myself, by spending time with them in my scenarios, always repeating tests, always with proper support, and always different scenarios, because, as a guy who tests for a living, there’s a saying: “Nothing can be potentially more misleading than an improperly done, or incompletely done, test”. I will spot check places like cameralabs for their distance tests, and if Roger has run wide open MTF on it, I absolutely check that because knowing the basics of what that test shows me allows me to make some reasonably educated guesses, but ultimately, it’s my testing. I will also add that I’m a firm believer that identifying a few trusted, known subjective reviewers whose findings you tend to agree with over time, who test thoroughly (not some quick cowboy in the camera store “test”) and looking at the consensus of those subjective reviewers will lead to better decisions about lenses than blindly following one of the many flat test chart sites. However, this approach takes more time. Thus, my advice is not to rely heavily on lens test sites utilizing test charts, for landscape or even architecture work.
What I have said and always saying about DXO and other useless chart testing might come off as rather controversial, but trust me, my approach was borne from wondering why certain lenses that didn't "test well" were great in real life. I tend to be the inquisitive type and don't mind the research, so what I learned, I learned. At the end of the day, I prefer people to make intelligent purchasing decisions, not ones that might have been misled. My quote again: "Nothing can be more potentially misleading than an improperly done or incompletely done test" - think about that for a second.
If a chart test has issues, which they do, then they need to be looked at with a grain of salt. Not ignored, but definitely not placed with such weight as many photographers apparently do. I'll take three or four subjective reviews from people I've read and trust over the years over a chart test every day.
If you are Sony user, then Fred Miranda's real life testing is the only one really reliable and unbiased testing site. If Roger tests more lenses stopped down then his source might be the best , however, he does not test many lenses stopped down , so I say Fred may be the best , then Roger next, after that probably Lenstip, and for infinity resolution, Cameralab tests can be good. All others are useless and I think should be dismissed.
UPDATE: Many people in many camera forums seem to think lenses should outresolve the sensor........and it's got be sharp at 100 percent pixel level to be any useful.
But the truth is lens should not outresolve the sensor:
Sensors should take advantage of lenses, so sensors should always have enough pixels for 100% pixels views to be very soft for a given lens. Everyone wishes for a sharper lens with a given sensor, but it is not always possible, and leads to artifacts if it is too sharp for a given sensor of the time, anyway. The imaging breaks up when a lens is too sharp for a sensor. It is the case for many FF cameras, still. Our 42mp sensor cannot outresolve most of Sony FF lenses released specifically for the E mount. In fact, even the cheapest ones do outresolve the 42mp sensor easily(at least in the center and mid zone).
If the lens coupled with the 108MP sensor is very soft, 108MP still has the advantage of creating lower pixel count output that is less aliased, with a more full color resolution than if you had started out with a lower pixel count to begin with (less so with quad Bayer, of course). All computational photography gets more accurate if you take full advantage of the resolution; the question is whether or not the handling and processing speed is then acceptable for the resolution now.
So I think the new Samsung sensor for their phones will be incredible, but is it the right time to introduce it now , is another question........and it is an issue of the current speed of their processing pipeline.
So many people hate high resolution bodies because lens cannot outresolve them is completely wrong, most of modern FF lenses do outresolve the 61mp Sony or the new 75mp Canon sensor easily. The real issue with the high resolution body for now is the current processor these camera makers use in their cameras cannot process big files very fast...
But as these are all friend companies of SoftBank, who now owns ARM holding , they can just order special CPU from them. I think it is the best solution for these cameras to become truly a smart camera. The bottleneck of the system is the poor quality cpu that most of these camera makers buying from Fujitsu or Texas Instruments.
UPDATE 2: Now it is really hard to recommend the Batis 40mm f2CF to any one even though I use one and like it quite a lot.
It seems just a bit overpriced. It should have been priced a bit more humbly..... Now people are not stupid. They do not just automatically pay 400 or 500 more for just the blue Zeiss badge like they did in the film era, unless they confirm it worth it in the long run.
It is sad to see this but the Zeiss seems to be a bit of an oddball like thirdparty lens maker these days.
Their position is hard to be justified. Their lenses (except some Otus) are optically worse than Sony's GM line lenses, Sigma's art series lenses, or some latest Voigtlander primes like the 65mm APO or the 21mm f1.4.
I wonder if it does not say Zeiss , who will pay more than 1000 US for any of these Zeiss Batis, Loxia or Milvus line lenses when an incredibly sharp lens like the Sigma 35mm f1.2 Art or Sony FE24mm f1.4GM can be bought for just 1500 US or less.
Maybe that's why their resell value is so bad , no one really wants the Batis, Loxia and Milvus. That seems like the sad reality for those of us long time Zeiss users.
If you need the sharpest prime in 35-50mm range , there is the Sigma 35mm f1.2 Art......nothing from Zeiss even the Milvus 35mm f1.4 match that kind of performance.
If you prefer a sharp but small travel 35mm f1,8 with super fast AF system, then there is the new Sony 35mm f1.8 , which seems to be a much more versatile lens than the Zeiss 40mm f2CF, as the AF of the Sony is much more reliable and bokeh seems a bit smoother(albeit with a bit more extra Lo CA).
So Zeiss really needs to wake up if they are still interested in consumer camera lens market.....but I have feeling they are no longer interested in ILC lens market.....
They can design incredible stepper lenses and other industrial lenses, so it is actually easy for them to design a set of incredible lenses for the E mount system, but they do not.
They try to sell these overpriced poor value lenses instead. I think this only means Zeiss is just doing it like a hobby business. It is really sad to accept it , but I think Zeiss is becoming an irrelevant player at least for the E mount community.
There was a long thread on this issue at Fred Miranda's site and many people actually disappointed with these Zeiss offerings for the E mount system.
Another serious issue with the Zeiss lenses is their brand name is still a bit more desirable than say Sigma, Samyang , etc.....but arguably less desirable than Sony GM, and much less desirable than Leica.
It's changed really for bad for Zeiss. The market rejects many of their lenses for the originally suggested price.
So many times Zeiss did super discount fire sells on many of their Batis , Loxia and Milvus line primes.......and still they are not moving......sad to see that.
UPDATE3: I find many people at DPR and FM criticizing
the Amazon and the BH price for the Sony 35mm f1.8 lens too expensive...and some of them blaming Sony for that high US price for that lens. But it is expensive there because of the tariff, not because Sony is overcharging for it.
Many of them seem to think it should have been 600 US , rather than 800US.
But, in reality, the original price of the lens ,without new Trump tariff , is actually only 64800 yen. And most of other market the price for the lens is similar to that. The USA price this time is very high for this specific lens due to the tariff.
So it may cost 800 USD in the US market, but it is much cheaper than that in Asia. So criticizing Sony for that US specific price is very silly. They should complain about the tariff, it is not Sony's problem.
And they can always order the lens from Amazon Japan or Hong Kong , then it will cost only a bit less than 650 USD.
I paid just 64800 yen at Map Camera here and it is including 8 percent sells tax here. If you are a foreigner here you can get it even cheaper as you do not have to pay the VAT here.
So those complainers at DPR also can get it for less than 650 US Dollars from Amazon Japan or who ever ships it to US from China or Japan.
And for 650 USD, I think it is a good lens especially for events or walk around street work.......even considered to be a huge bargain lens.
It has incredible new AF and very good anti-flare coating.
It is also very sharp even wide open at least in the central area and about 2/3rd of the frame..the corners are a bit soft wideopen, but for shallow DoF stuffs sharp corners are not really needed many times.
It is really sharp across the frame at f4 and on.
Distortion is also well controlled, a very low barrel type of distortion.
As DPR points out, Axial and Lateral CA are a bit excessive on this lens and can be a serious issue, but it is easy to correct with any software, the C1 Pro seems to do the best here but even ACR does fine.
Many people who have never shot with this lens hate it for its busy bokeh and clinical look.....and I agree its bokeh is not excellent, but still it is not as bad as they say it is. As Rishi said in his review of this lens, it produces smooth OOF area most of times. Only rarely produces terrible nervous bokeh.
What is really funny about those DPR trolls and FM lens snob cult members is that they hate this lens for its so called NERVOUS bokeh and many of them compare it to much more expensive and huge Sigma 35mm f1.2 Art...
They trashed the Batis 40mm f2CF for the same reason once, and they also concluded it is not as good as the Sigma 40mm f1.4 Art. Many called the Batis 40mm f2CF too clinical, just like they say about the new FE35mm f1.8.
But, many of those who are in the market for the Batis 40mm f2CF or the FE35mm f1.8 are not actually interested in the huge Sigma , or they may be interested in the Sigma 35mm f1.2 Art but also interested in the tiny Sony or Batis for street stuffs.
So it is really ridiculous to compare the Sigma Art 35mm f1.2 o the new tiny cheap Sony. They are designed for different kind of user groups.
Still, those FM lens snobs compare it to the Sigma and trash it based on someone else's images made with it. I find it very dishonest and very unfair to the Sony 35mm f1.8.
One guy with the Sigma 35mm f1.2 Art told a guy wanted to get the Sony or the Batis for his kids photography he should not get it because his kids will be reviewing these images of them and disappointed about the bokeh of the Sony or the Zeiss 10 years from now.....
Well you can hate any lens , but it does not sound logical, a very strange reason to hate a lens. 10 years from now his kids will be disappointed about the bokeh, serious?
Anyway, I found those lens snobs really dishonest and extremely misleading with their silly lens dogmas.
I mean if a lens cannot focus tracks a kid in lowlight , what is the point of a bit more creamier bokeh or a bit more smooth or artistic rendering?
UPDATE4: Now Roger Cicala at LensRentals tested the FE35mm f1.8 and his results seem to be really positive. I think his results positively confirmed that DPR was right at least on that lens and all the members of the rendering cult at FM forum were wrong again.
The FE35mm f1.8 seems to be a superb lens with a set of amazing resolution figures to support that.
I was really glad I trusted my own test and DPR, not the members of the lens rendering cult.
Roger also confirmed that the FE24mm f1.4GM is indeed the best 24mm f1.4 lens ever made for any mount system(however, it is not the best 24mm lens or sharpest 24mm prime ever).
On this one , the members of the rendering cult were right, however , if you compare it to the Zeiss Loxia 25mm f2.4 or the Sigma 28mm f1.4 at f5.6 and on the Zeiss and the Sigma are sharper than the Sony FE24mm f1.4 GM.
So if you do not need f1.4 speed at 24mm , then there are better options than the GM, but if you absolutely demand that speed at 24mm , then there is no better choice than the GM.
So congrats to Sony and us FE users. Sony seems to have been on a long roll with these new FE lenses.
To be honest , none of their newer lenses released after 2016 are bad. Most of these are actually excellent lenses with much improved QC.
UPDATE5: Now Zeiss Batis series lenses are really doomed.
IMHO, there will be no more Batis lens announced very soon as both Zeiss and Tamron seem to be losing quite a lot of money on that series.
I think the Batis 40mm f2 CF lens is a real tough sell lens now just like their Batis 85mm f1.8 has always been since the FE85mm f1.8 announcement in 2017, and many of us honestly regret buying the 40mm f2CF. I know I will lose quite a bit when I sell it in next month.
It is a fine lens, but not really great one especially for the rather high price of 1299 USD Zeiss is asking for it.
It should be a 800 US lens. Other wise, it cannot compete with the lenses like the Sony FE35mm f1.8, the Sigma 35mm f1.4 Art , 35mmf1.2 Art and the Samayang 35mm f1.4.
There are many other cheaper and optically as good or even a bit better options in this focal range now, and most of these are either faster or sharper than the Batis 40mm f2CF.
Honestly, if I were buying a 35mm -40mm moderately fast prime now, I would never even consider the Batis 40mm f2CF lens. I think I would just wait for the Tamron 35mm f1.4 SP lens.
Now Tamron makes it for only the D-SLR systems Nikon F and Canon EF. But I am sure they will make FE version of that lens and I think that will be their next big announcement at CP+ show in next Spring here.
UPDATE6: in the last December , I said I would sell the Batis 40mm f2CF and I did , but I bought it again.
Initially, I thought I 'd get it and test it against the Voigtlander 50mm f2 APO, and then just sell it or return it.
But I decided to keep it , it is actually a very sharp lens, even rival to a good copy of the Voigtlander 50mm f2APO.
Now I think the Batis 40mm f2CF is the most underrated E mount lens. The 40mm and the 18mm Batis lenses are really underrated.
UPDATE7: Many forum lens snobs trash zooms even a great zoom like the FE16-35mm f2.8GM in favor of a set of heavy fast primes like the Sigma 14mm f1.8 Art, the Voigtlander 21mm f1.4, the Sigma 35mm f1.2 Art, the Sigma 40mm f1.4 Art,etc. Sure these lenses are great in terms of sheer resolution and over all image quality at wide f stops, but as a part of a good travel kit or a street kit , are these heavy primes still great or even practical?
The huge Sigma lenses are not at all interesting to me , I can accept the size of my Sigma 28mm f1.4 Art, but the 35mm f1.2 is just a bit too much for a prime. I cannot fly with just one 35mm prime , that means I will have to carry on a couple or more lenses in addition to that lens with me and that may cost me a lot of extra money as I need to upgrade my flight to a business class.
Sure the Sigma 35mm f1.2 is a great lens for pro portrait shooter, but for most of us the Sony FE35mm f1.8 E is a much more practical lens, and IMHO, also a bit more versatile lens.
I think these moderately sized practical lenses like the Sony 35mm f1.8, the Batis 40mm f2CF, the Sony FE85mm f1.8 and the Batis 18mm f2.8 , even the Sony FE16-35mm f2.8GM(or the even more compact Tamron version) are all underrated because those online camera forums are dominated by the members of the super fast lens worship shallow DoF cult.
But if you often shoot your lenses at f5.6 o f8, then those super fast primes are really a big waste of money.
I mean most of super fast lenses become sharpest at f2.8 -f 4 range and even at f5.6 they are not very sharp any more.
If you follow Lenstip.com lens resolution tests , you may know what I am trying to talk about here.
Most of the Sigma Art or Sony GM line primes are designed to really shine at f1.2 or f1.4 and they are not really great at f8.
The Loxia , Voigtlander , and even cheaper Sony f1.8 primes are usually better than the similar focal length super oversized Sigma Art lens at f 5.6 and on.
If you compare the FE85mm f1.4GM vs the Loxia 85mm f2.4 or the Batis 85mm f1.8, or even the FE85mm f1.8 E at f8 or f10, you know what I am talking about here.
The FE85mm f1.8E is sharper at f5.6 and f8 than the GM.....all computer based lens tests and Cameralabs infinity resolution test have proved that.
