testing software
28/01/09 20:24 Filed in: Personal
A camera body, certainly a mechanical one with only a few controls for manipulation, takes a few months to a year to test it thoroughly.
A current digital camera with hundreds of combinations would take at least the same amount of time to test. But you will find comments (I would not dare to say that is it is a test in the classical sense) on cameras after a day of handling and some lab checks.
A lens is relatively simple to test, as there are only a handful of characteristics to analyze, but even in this case a really meaningful test would take several months.
A film with several developers would take on average several weeks to test, given the use of a densitometer.
A software program can take any time period from a month to a year to test. Most postprocessing software or developing software as it is sometimes called, are very complex programs with hundreds if not thousands of combinations.
I am very surprised that one sees everywhere reports that make strong assessments about the quality or performance of a software program, like Photoshop or Lightroom or Aperture.
I am working in the world of software and I know that testing a piece of software is a very specialized and disciplined activity. You need test protocols, controlled test cases and a very detailed suite of actions that can be executed several times without introducing new parameters.
I am using the following post processing software:
My personal favorites are Aperture, PWP and Silverfast. Why?
I use only a very small selection of tools in these programs. I load the Raw file, do a minimal of manipulations on the gradation curve and unsharp actions (identical to what you can do in the classical darkroom), convert to BW and export as TIFF. These programs support my workflow very well and produce excellent TIFF files for printing. Here I use the Epson 3800 with several types of Hahnemuehle papers.
The number of steps you have to take and the many options per step are mind boggling.
I have found by trial and error a series of settings that bring me the results I want and am pleased with.
This result however cannot qualify as a test. A true test would imply that you define at first what the results are that you expect and what manipulations can alter the result and how, given all the parameters that can influence the result. Preferably these expected results should be defined in quantifiable units. I am only using a few possible controls and assess the result visually. A report by me about all these programs and the achievements would have very limited value. That is why you will not read on my website about tests of software. A truly valuable test would demand such an amount of preparation and execution that is impossible to dedicate to.
What you can find on all tests in the internet is a very simple comparison of an image file, processed by a program with a set of parameters selected and fixed. What we do not know and that would be the Big Question, is if we can be certain that the selected parameters and options really do represent the best the program can do given the characteristics of the image file. We simply do not!
Are we really sure that the selected options are the optimum and that the sequence of actions is the best solution?
Frankly speaking we do not know!
So-called tests are then simply snapshots of what one operator can do given limited experience with all options and parameters and the interactions involved.
In mountain biking or car racing or even photography the rule holds that an experienced user with moderate equipment can achieve more that an unexperienced user with high end equipment.
The situation with the post-processing software is identical: if you know how a program behaves you can get better results than when you randomly select a few features.
A very basic point may illustrate this: every program has the sharpness control where you have to select radius and amount. But every program gives different results on the same picture.
How do you value this conclusion? Presumably I can get identical results with different settings. Hw do I conduct a test with controlled parameters that expose the true differences.
I would say this is a very hard act to execute.
I have not read any software test ( in the image file processing software) where these aspects are addressed in a satisfying way.
My current conclusion is that all tests of software programs lack the qualification of "test" and are at best informed reports about personal experience and settings. Quite valuable but not good enough to support the selection of a program as good or best.
A current digital camera with hundreds of combinations would take at least the same amount of time to test. But you will find comments (I would not dare to say that is it is a test in the classical sense) on cameras after a day of handling and some lab checks.
A lens is relatively simple to test, as there are only a handful of characteristics to analyze, but even in this case a really meaningful test would take several months.
A film with several developers would take on average several weeks to test, given the use of a densitometer.
A software program can take any time period from a month to a year to test. Most postprocessing software or developing software as it is sometimes called, are very complex programs with hundreds if not thousands of combinations.
I am very surprised that one sees everywhere reports that make strong assessments about the quality or performance of a software program, like Photoshop or Lightroom or Aperture.
I am working in the world of software and I know that testing a piece of software is a very specialized and disciplined activity. You need test protocols, controlled test cases and a very detailed suite of actions that can be executed several times without introducing new parameters.
I am using the following post processing software:
- Picture Window Pro,
- Lightroom
- Bibble
- Photoshop CS2 and ~Elements both with Nik Efex and Sharpener
- Capture One
- Aperture with Nik Efex and Sharpener
- Light Zone
- Silverfast DCpro
- Sylkipix
- some more
My personal favorites are Aperture, PWP and Silverfast. Why?
I use only a very small selection of tools in these programs. I load the Raw file, do a minimal of manipulations on the gradation curve and unsharp actions (identical to what you can do in the classical darkroom), convert to BW and export as TIFF. These programs support my workflow very well and produce excellent TIFF files for printing. Here I use the Epson 3800 with several types of Hahnemuehle papers.
The number of steps you have to take and the many options per step are mind boggling.
I have found by trial and error a series of settings that bring me the results I want and am pleased with.
This result however cannot qualify as a test. A true test would imply that you define at first what the results are that you expect and what manipulations can alter the result and how, given all the parameters that can influence the result. Preferably these expected results should be defined in quantifiable units. I am only using a few possible controls and assess the result visually. A report by me about all these programs and the achievements would have very limited value. That is why you will not read on my website about tests of software. A truly valuable test would demand such an amount of preparation and execution that is impossible to dedicate to.
What you can find on all tests in the internet is a very simple comparison of an image file, processed by a program with a set of parameters selected and fixed. What we do not know and that would be the Big Question, is if we can be certain that the selected parameters and options really do represent the best the program can do given the characteristics of the image file. We simply do not!
Are we really sure that the selected options are the optimum and that the sequence of actions is the best solution?
Frankly speaking we do not know!
So-called tests are then simply snapshots of what one operator can do given limited experience with all options and parameters and the interactions involved.
In mountain biking or car racing or even photography the rule holds that an experienced user with moderate equipment can achieve more that an unexperienced user with high end equipment.
The situation with the post-processing software is identical: if you know how a program behaves you can get better results than when you randomly select a few features.
A very basic point may illustrate this: every program has the sharpness control where you have to select radius and amount. But every program gives different results on the same picture.
How do you value this conclusion? Presumably I can get identical results with different settings. Hw do I conduct a test with controlled parameters that expose the true differences.
I would say this is a very hard act to execute.
I have not read any software test ( in the image file processing software) where these aspects are addressed in a satisfying way.
My current conclusion is that all tests of software programs lack the qualification of "test" and are at best informed reports about personal experience and settings. Quite valuable but not good enough to support the selection of a program as good or best.