The Online Photographer

Check out our new site at!

Saturday, May 26, 2007

On Lens Reviews

I appreciate Erwin Puts' comments in his recent article "On Lens Reviews," and I think he's on to something. His clarification of the various approaches to lens reviewing is right on, and better articulated by him here than I've seen from other writers elsewhere.

He's got me pegged, for one thing, when he says that my approach "takes the whole imaging and viewing chain as an integral process and reviews a lens for its impact on the presentation of the scene as fixed on a print and viewed by an observer." I couldn't have said it better myself, and it's very true—the proof for me is the print, and what you can't see in the print doesn't count for very much as far as I'm concerned.

In other words, I'm an "eyeballer," as Phil Davis used to call me (not very approvingly)—although I'll register my usual protest, which is to say that I think I'm a good eyeballer.

But the empirical, practical approach is in some ways not very descriptive. In fact, I don't even like to call my lens reviews "tests"—I prefer the word "trial," because all I'm doing is trying the lens and then describing my results to the reader. A "test" implies a scientific approach that is experimentally sound, measurable, and repeatable. Erwin Puts names that approach in honor of Geoffrey Crawley (Editor of the British Journal of Photography for a 21-year tenure), but it could just as accurately be called the Erwin Puts Approach.

"The Crawley-Puts Approach" evaluates the technical properties of the lens in isolation, without even muddying the waters with the contributions of the imaging substrate (film or sensor), much less all the other elements of the imaging chain. The Johnston Approach, as Erwin names it (I'm flattered, even if I'm not sure he means it as flattery!), has the advantage of being more practical, and the disadvantage of being more limited. For instance, I usually include in my lens reviews disclaimers as to what they will not shed any light on—starting with color transmission, since that's largely invisible to the black-and-white films I normally use.

The much more rigorous Crawley-Puts method of describing a lens is ultimately more accurate, as well as more readily applied to differing applications, but has the drawback that it might not be descriptive of what users will actually experience using the lens for their own work. Indeed, this is often reflected in Erwin's writings when he notes that high levels of technical skill are necessary to extract the very best out of any particular lens (and sometimes to detect the differences and distinctions he describes). This is a repeated refrain in his writings, and it's obviously something that concerns him.

Armed and dangerous
The key to writing a good subjective review—the kind I write—is to arm the reader with as much information about the terms and conditions of the review as possible. My recent controversial pair of posts about the Leica M8, for instance, were widely criticized. Some criticisms were simply errors or were based on errors (one commentator concluded that the vehicle in the first picture in the reviews, of a parked BMW SUV, must be my own car. He then used this premise to speculate about my feelings toward German technology. (I drive a Ford.) Another commenter said that I work for Apple, and another stated that I write for Outdoor Photographer magazine. Neither of those things are true either). But many people made criticisms that were very valid. Even certain friends complained that I hadn't done enough shooting with the M8, for instance. I ran into repeated references on other forums saying I had shot only "90 pictures" or some number close to that. Actually what I said was that I filled up most of a single 1GB SD card, which might be 200 or 300 shots when you consider the ones I deleted as I went along—careful reading without assumptions is still required. Regardless, the underlying criticism that I hadn't shot enough with the camera still holds.

But how did they know that? They knew because I told them, that's how. I'm always amused, after I write a subjective review, how many people raise objections using the very information that I deliberately provided for them in the text of the review.

That's how a subjective review works best, in my opinion. Readers should be well informed of the writer's prejudices, the extent of his trials, the conditions under which he worked, even his tastes. They can then take all that information into account when evaluating and applying—or, yes, dismissing—his conclusions.

Called for and needed
The Crawley-Puts Approach admits of no such "slop." There is little room for impressionism. When a technical measurement is made, it must be made exactly: it's not good enough to say "Well, I wasn't being very careful, so maybe the optical resolution of the aerial image two-thirds out from the center is 10 or 20 lp/mm better or worse than my figures show." No. That won't do. Exactitude and rigor are called for, expected, and needed.

In the end, with this approach, what the reader is left with is a description only of the lens and its potential performance, and there is no guarantee that he will see this potential realized in his own work. But the description he does get is complete, thorough, and exact—more so than a subjective trial could be.

There is room for both approaches in reviewing, of course.

I've said nothing about what Erwin calls "A Third Way," but I recommend that you read his comments carefully. What he describes there is, among other things, one of the fatal downfalls of most high-end hi-fi reviews.

A final point could be raised, which is that it is very rare for any single review to be 100% of one type or the other. I, and other subjective reviewers, sometimes employ technical measurements to discover or support a point of argument. And, even in a technical review, tastes and personal judgments will sometimes be found. It is perhaps understandable when a subjective reviewer is not as careful as he should be with technical measurements, but that's no excuse. And it might also be understandable when a technical reviewer is influenced by personal tastes or previous experiences yet casts these things as facts, but he should be vigilant against that error. We all try, I'm sure.

The timing of this article is fortuitous for me, too. I'm about to embark on a new review project, and these discussions make a good foundation for that review.



Blogger John Roberts said...

"it's very true—the proof for me is the print, and what you can't see in the print doesn't count for very much as far as I'm concerned."

I agree wholeheartedly! Even though I only shoot digitally now, the print is still the thing. Images on a computer screen, even those at 100% magnification, aren't the real world for me. Only the prints matter.

With that in mind, your approach of a lens "trial" is more helpful to me than a lot of numbers cranked out by a "test" in an optical lab.

3:31 PM  
Blogger Scott Kirkpatrick said...

When I read Erwin Puts' essay, "on Lens reviews" a few days ago, I didn't realize at first how offensive it must have felt to Sean Reid, who is thinly disguised as Donald Duck, in the section which characterizes a fourth way of reviewing lenses. Apparently the fourth way consists of saying things already known to Mr. Puts, but "only recently discovered by

I write scientific papers for a living, and I see something like this in scientific credit assignment and instant judgements. In some areas where I have worked, I get more credit than I deserve, and in others that I get interested in I am infuriated to be dismissed as a mere newcomer to a field fondly believed by its life inmates to be so subtle that only a
long apprenticeship (at their feet, of course) would qualify anyone to speak aloud.

So Sean should quack proudly -- Erwin has noticed him, and been stimulated to produce his own summary review of the two 75 mm lenses. Mr. Puts' review, I thought, had both interesting facts about the design approaches taken and revealing insights into his set of values. He cares about the performance of a lens across a range of system requirements not present in any single camera, across the whole 35mm full frame, and for levels of detail that are seldom found in real images, and never resolved when shooting hand-held. It warms his heart to know that a lens has solved a greater number of these problems than has any other lens, and this lens will sit proudly in the "classic" section of his camera
bag. His desk is apparently covered with MTF data taken on an optical bench with a microscope as imager, not in any camera. The data he sometimes shows is not the data published in Leica spec sheets, so he must take his own measurements or have friends with
those tools. I'm interested in that information, and the rest of his pontification either amuses me or escapes me altogether.


5:50 PM  
Blogger Mike Johnston said...

Sean Reid did not even cross my mind in connection with the Donald Duck joke and the "Third Way." Not even faintly. Sean's reviews are much more subtle and intelligent than that. I suppose only Erwin can say for sure, but I thought he was talking about amateur lens comparos of the sort you see all the time on the dpreview forums and other such forums.


5:54 PM  
Blogger Scott Kirkpatrick said...

The Donald Duck reference leads off the paragraph just below the "third way" discussion of web-based pixel peeping. This paragraph contains Puts discussion of the comparative designs of the Leica 75/2.0 Summicron and the Cosina-Voigtlaender 75/2.5, which Sean had just reviewed. Sean found that the CV lens was sharper than the Leica across the field, but less contrasty. Puts explains, with some interesting details, how that might come about. I don't think he would have bothered to write up that comparison, had he not been irked by Sean's review.


6:37 PM  
Blogger JanneM said...

Mike, would you be comfortable with substituting "final print being viewed" with "end result put to use" as a description of this approach to reviewing? In that case I'm fully on board with the validity of the approach.

8:14 PM  
Blogger Mike Johnston said...

I learned a clever move from one of Alan Watts' favorite latinate footnotes--the footnote was "mod. auct." which must have stood for "modus auctororis" or something close to that--he used it to mean "modified by the author" or "in the mode of the author." So you'd just use the quotation with your own wording inserted and cite "Erwin Puts mod. auct."

I miss Alan Watts!


8:26 PM  
Blogger Sean Reid said...

Mike wrote:
"Sean Reid did not even cross my mind in connection with the Donald Duck joke and the "Third Way." Not even faintly. Sean's reviews are much more subtle and intelligent than that. I suppose only Erwin can say for sure, but I thought he was talking about amateur lens comparos of the sort you see all the time on the dpreview forums and other such forums."

Hi Mike,

Thanks very much.



9:51 PM  
Blogger Player said...

The First Approach is like subjective creative writing, poetry, fiction.

The Second Approach is like objective prose, reporting, non-fiction.

The Third Way is like pseudo-science, pop-art.

3:02 PM  

Post a Comment

<< Home