It is not great news to note that any lens for photographic purposes has defects. There are intrinsic optical defects (the results of the optical design and glass choice). There are mechanical defects, some intentional (the construction of the mount generates flare) and some non-intentional as the drift of tolerances. And we have the occasional production failures which occur and come to light only after it has happened.
How good a lens may be, you will always find characteristics that are not good. Every retro-focus wide angle lens will show softness and flare in the corners and edges of the frame, not only wide open but also stopped down. This is a basic trait of the design. Symmetrical designs, like the Angulon are indeed better in this respect, but show other ‘defects’. It is now customary to view the images at 100% or even 400% at the computer screen and search for characteristics that attract visual attention. Persons who take landscape pictures with a 21mm lens on 135-format and detect flare or softness in the edges of the image, should not complain that the lens is wrong or not well adjusted. This behavior is state of the art, so to speak: it is part of the intrinsic design. The exaggerated expectations of many photographers are fed by the industry that claims to have lenses that are ‘tack-sharp’ wide open over the whole image field, extending into the far corners of the image. And of course there are the countless websites and statements in the innumerable internet forums that attach mythical claims and unsubstantiated allegations to lenses. As soon as somebody notes that he, after testing is disappointed with a a certain lens, some one else will immediately jump up to claim that his version is beyond reproach.

The best arbiter in such disputes is the MTF graph. When done correctly this information gives the true state of a lens. And looking at graphs of wide angle retro-focus lenses we see always a significant drop in the contract at the edges, mostly beyond the 18mm image height (based on a 21.6 diagonal of the frame). Discussion closed you would say. In a sense it indeed is. But the MTF does not tell you how the representation of the detail in the edges is: is it a blurred image, is there flare which reduces contrast? What exactly can we see? Here lens designs differ and this cannot be inferred from the curves. You really need to look at the pictures. But then another serious problem occurs. How can you create a test setting that really is accurate for the task at hand. A simple example: it is very difficult to align the camera/lens exactly parallel to the the object you wish to photograph. If it is not parallel, the behavior of the edges will differ and which one is representative, if at all?

For the tests of the S2-lenses I worked with a lab setting where the test targets are absolutely plane and aligned and the camera/lens is adjusted such that the sensor plane is absolutely parallel to the target plane. (this is one of the few instances where Live View is a really good thing). To get the best results, accurate focusing is a must. I can confess that finding the best focus is not easy and takes much time and concentration and trail and error to find the optimal focus. I can also note and that should not be news, that no AF system now on the market sets the focus accurately. Even the redoubtable Nikon D3x would not find the optimum focus position: manual adjustment was required. The same goes for the S2 and the Sony A900 which were used also to get a good range for comparison purposes. The M9, to complete the picture, also required focus bracketing to find the optimum position. And it should be stated in red, so that the reader gets the message, that very small changes in the focus setting/movement have serious consequences for the image quality.
The generation of the MTF graphs is relatively easy and always very accurate. To get the optimum position the program moves the light source to find the best focus plane (smallest point source or slit width) to micron-accuracy. Then you know that the best quality is assured.
This measurement produces accurate results within its systems setting. There is no relation to the practical photographic situation where all kinds of degradation effects can occur: AF errors, range finder alignment, mechanical tolerances, the observers eye and so on. There are three conditions that are never mentioned in photographic testing that have dramatical impact on the performance of the system: (1) handholding the camera will reduce the optimum quality by at least 50% (image stabilisation really helps!); (2) you cannot hold accurate focus when working without a tripod: the AF is generally not reliable and even the slightest movement from the head or body that moves the camera from or to the object during or after focusing will shift the optimum plane of focus; (3) the usual objects to photograph are three dimensional with many differently angled surfaces and a large depth. The behavior of the lens when recording out of focus areas before and after the plane of sharpness (wherever that may be located) differs from design to design and from distance to distance. And the way the lens reacts to specular highlights and diffuse highlights (all of these effects lumped together under the heading of bokeh) is not seen in the shape of the MTF graphs.

These effects are however of prime importance for the image composition and visual impact of the photograph. The current trend in photographic testing is using a slant edge or one of the many test charts found on the internet and go for a kind of simplified MTF testing. A program as Imatest does a good job here, but one must realize that finding a numerical value at 50% contrast or a figure like 2134 lp/ph does not say anything important about the photographic behavior of a lens. We are in danger of cutting off one’s nose to spite one’s face! Because lenses are ranked according to these simple figures and numerical results, optical designers are forced by the marketing departments to optimize designs that reflect these preferences.

There is arising a big and widening gap between the constructs of optical designers and the serious photographic practice.
Many optical designers of previous generations were actively involved in photography and could understand what effects a certain design and its inherent balance of errors might have on the resulting photograph. Modern optimization programs are blind to photographic demands.
The two S2 lenses I have checked, the Summarit-S 2.5/70 and the Apo-Elmar-S 3.5/180, show excellent respectively superb MTF behavior. In this case the results ‘in the field’ do reflect the performance as implied by the curves. But there is more: these two lenses show outstandingly good color correction, above what is seen in the classical 35mm scene. The lenses are corrected even for the deep blue part of the spectrum (in the region to 400nm) where the contrast values are still above 90%. The very smooth definition in highlights and the faint occurrence of color fringes at edges of strong contrast are the result of this correction strategy.

Regardless of the quality and relevance of the testing procedures and methods, we should realize that many test-results as can be found in the internet and in printed magazines too are not relevant for photography. The second important observation is this: even if the results are relevant, we can hardly translate these results to our normal photographic practice. When we restrict ourselves to studying test charts at 400% on the computerscreen or looking for defects in the smallest posible details, we are fooling ourselves. As soon as we want serious pictures with visual impact, we need more and different information. And perhaps we need lenses that are designed with a new paradigm, which in effect is an old one: what is really needed for high impact photographic images.

We might argue that all of this is nowadays hardly important as we have lots and lots of potent software that can correct most of not all imaging defects. There is a grain of truth here, but any correction will change, degrade the image or shift the visual balance.
Images made with the Nikon D3x and the Hasselblad with 39Mb digital back show remarkably small differences in performance and image quality on a calibrated large computer screen. One must acknowledge the very advanced and potent software of the prime Japanese camera builders. This performance compensates a deficit of at least 10Mb of pixels in the sensor size or a substantial difference in optical quality.

The main conclusion however is inevitable: most test results are based on a small subset of performance criteria that are not really meaningful for the photographic practice nor truly representative of the really important parameters of a modern lens design.
The marketing machine of the major companies, the reports by magazines and websites where lens performance is described in cloaked terms and the over-enthusiastic claims by users have created a fantasy world of expectations for optical excellence, in the same way as Photoshop manipulations have created a fantasy world of feminine body shapes.

Long ago, a marketing man of a major 35mm camera manufacturer admitted that he used promo pictures made with a medium format camera. This does not happen anyone, one may presume. But we should seriously reflect on our current lopsided approach to image quality and find a new avenue to intimately connect photographically important image quality with quantitative test results. As was the case in the past when the first MTF graphs of photographic lenses were published.

The relevance of Aristotle

Aristotle lived 2500 years ago, but most of his thinking is still relevant today. One of his important ideas is the theory of money. Money is in itself worthless, but when used as a means to an end it is valuable. What you buy must be worth more than the money itself, otherwise you would not wish to buy it. The product you buy should give you a certain positive feeling that you value more than the product itself. And at the highest level the feeling you get from the product should make you happy and help you live a good life. That is the goal for all of us: live a good life and be happy.
The amount of money and the amount of products we have or own is limited by the definition of a good life. The important question then is to define what a good life is, because that definition determines what we need and what amount of money we should have. This approach has a dangerous connotation: when the idea of a good life vanishes, there is no limit to the amount of money we want to have. How much money is enough? The answer is simply that the maximum is limitless, just as the number range does not end, even in infinity. As soon as the idea of a good life disappears, we are stuck in an endless quest for maximizing our needs. The logical consequence is greed as we see in the Wall Street inhabitants.
Our culture has lost the focus on the good life and therefore we want to maximize every experience and need: the idea of maximizing our needs defines our working life, sex life and all of our other experiences.

It is very easy to map this thinking on the current state of photography. The idea of a good life can be interpreted as the idea of a good photograph or in modern parlance a good image. What the content of such a photograph should be is for now not important: it might be a fully photoshop-distorted visual construct. The main question is this: what product do we need to create this kind of image that makes us happy. The derivative question then will be how much money do we need to buy that product.

Sometimes browsing internet forums, I am always surprised that individuals sign their contribution with a listing of the products they own, as if this adds weight to the opinion they express or adds importance to the standing of the person. It would be more interesting to show important pictures that they are happy with: the quality of life so to speak.

The two golden ages of photography were defined by the quality of the images that were being made. There were cameras that made these pictures possible, but mainly it were the photographers who created the pictures and defined what good photography should be.

Currently manufacturers of cameras follow the maximizing approach: more is better and more makes you happy. The sad fact is that most buyers of equipment are deeply insecure about their purchase: the reviews by sites and magazines who live by delivering test results are mostly focused on making the buyer happy with his purchase. The weak point is the lack of a reference to the definition of a good life/photograph. If we are aspiring to create good photography we select by logical thinking the camera/lens system we need to accomplish our goal and then we are immune for advertising or other mind bending attacks to influence our feelings.

For several weeks now I am very busy photographing with the Leica S2. My first inclination was to go for the maximizing approach: compare the S2 feature-set to what others have to offer. But then I changed my mind and went for the Aristotle analysis: would I be happy with the S2 and will this product help me to create the pictures I would define as good photography.
In fact this is the idea that Leica has followed since the construction of the Leica Null Series: a product that makes you happy and helps you to make pictures that are good in emotion and content.