Test method: Image quality of a camera: tests supported by measurements

Resolution, noise and autofocus. We measure these important camera features according to the same strict protocol for each camera. We combine the measurements into graphs and tables, which you can use to compare the image quality of cameras. Initially, these numbers and graphs might be overwhelming. And you may wonder how those numbers can help you choose a good camera.

camera review method, Imatest, camera image quality, camera noise, autofocus accuracy

We are convinced that it is important to measure image quality, because measured values are directly related to what you'll see in practice. We know from experience that - regardless the camera a picture has been taken with - whenever the noise is less than 2.5%, an A3+ photo print can be made without the noise as a disturbing factor.
A trained eye can distinguish a color difference (delta E94) of 2, but an average eye will only see a color difference (Delta E94) of 6. Why worry if the average color accuracy of two cameras differ by 1 Delta E? You won't see the difference in real-life pictures. On the other hand, the color accuracy for specific colors, like skin tones, might be worse than the average color accuracy. Which perhaps leads you to calibrating the color rendition of your camera.
Measurement results and graphs have, therefore, a predictive value. Whenever possible, we will show images in our reviews that - like numbers and graphics - will help you to determine what lens or camera is best for you in terms of price and quality.

Sensor resolution

For the resolution measurements we use the Imatest software. For a camera resolution test, we select a lens with the highest possible resolution, and use it at the optimum aperture. As an illustration, the result of resolution measurements of a number of jpg files from the Canon 5D MK2, measured at different ISO settings, is shown here. The measurements are given in the form of bars. The higher the bar, the better the camera performance. To the left of the bars you will see a scale with a color gradient that will help you interpret the numbers. The color gradient ranges from red (poor) to green (good). As a rule of thumb, camera RAW files will give you higher resolution than jpg files, although the difference will not always be distinguishable in practice. camera review method, Imatest, camera image quality, camera noise, autofocus accuracy

In order to minimize possible negative effects of the lens on the performance of the camera as much as possible, we measure the resolution in a camera review only in the center. This choice leads to differences between the resolution measurements shown in the lens reviews, where the resolution in the corners and the edges of the picture are also taken into account taken (see: How we test lenses ) and the resolution measurements shown in the camera reviews.

Because the resolution of an image depends on the image processing, we measure the resolution of a camera in two ways:

  • Resolution of jpg files which have been saved in the camera, using camera settings that will minimize processing (sharpening, contrast, color adjustments) of the file. We keep the camera settings as constant as possible for each brand. The settings for jpg files sometimes vary considerably, depending on the camera preset and the camera brand. Every single camera brand offers several choices for jpg files (Natural, Standard, Faithful, etc.). Therefore, it is tricky to compare the resolution of your camera with the resolution of other camera brands on the basis of the measured resolution jpg files. The values of the jpg files are fine to compare the performance of different lenses, tested on a camera body of the same brand. Comparison of the resolution of a JPG file with the resolution of a RAW file, on the other hand, gives you a good impression of the sharpness impression of the JPG files that are stored in the camera.
  • Resolution of RAW files from all cameras by converting them in Lightroom in exactly the same way. We do not use any noise reduction while processing the RAW files. Thus you maintain the highest possible detail in the recordings (3000 LW / PH), while the noise, though measurable, will not be visible in print. In general, the sharpness impression of RAW files after development is higher than the jpg files. Comparing the noise in these RAW files with the noise in the jpg files, gives an indication for the amount of noise reduction applied at in-camera generated jpg files. The measured resolution of the RAW files is a better indicator (in stead of jpg resolution) for the comparison of different (brands of) cameras.

What can a measurement tell me?

camera review method, Imatest, camera image quality, camera noise, autofocus accuracy rawijsvogel

For each camera feature we strive to show you what the measurement results mean for the daily photo session. Here you see the same Kingfisher, on the left a jpg file with a resolution of 2000 lines per picture height (LW / PH) and on the right a RAW file with a resolution of 3000 LW / PH.


rawnoise  noise-2

We measure camera noise using Imatest. The amount of noise in a gray scale is expressed as a percentage. The higher the reading, the worse the picture quality you will obtain because of noise. We show you pieces of a gray card where you can see with your own eyes how a measured amount of - for example - almost 4% (at ISO 6400 in the above picture) is related to a measured noise of 7% (ISO 25.600 in the above figure). The jpg files perform in-camera noise reduction, and thus seemingly score better in terms of noise than RAW files. As we have explained earlier, this is not true.
By comparing noise in RAW files with the noise in jpg files, you get an indication for the extent to which camera noise reduction has been applied in the JPG files. The noise measurement of a RAW file we report can be viewed as a worst case. If you apply contrast, sharpening and noise reduction carefully, while developing RAW files, you'll likely to see less noise in your RAW images. Whether the amount of noise in your processed RAW files is less than in JPG files depends on many parameters such as the brand and type of camera, the software you use and your own photo editing skills.

Color accuracy

ICE (International Commission on Illumination) has devised several models, which describe colors and color differences. We measure the color difference "Delta E94", where the E stands for the German "Empfindung" (loosely translated: sighting). The measured Delta E94 is (unlike "Delta AB") corresponds with our observation: a twice higher value means a twice as big a difference to our eyes.
This figure shows the color accuracy of a camera under artificial light (the circles) compared to the ideal value (the squares). The further a square is removed from the corresponding square, the greater the color difference between the camera and reality. The further away a square or circle lies from the center, the more saturated ("brighter") is the color. In this example the camera shows a strong red cast (all circles are shifted towards red in comparison with the corresponding right squares)
minicanon-5d-mk2-flits-kleur 6866 colorerror
Another way to show the measured color differences, is a color chart where the upper half corresponds to the reference color and the bottom half represents the camera color. Along with the color chart, we summarize the results.
The smallest color difference between two brightly colored squares side by side which a human eye is able to distinguish, is about 1 Delta-E94. But the human eye is more sensitive to color casts in gray level and midrange. In the gray scale, an untrained viewer will already notice a difference of 0.5 Delta-E94. Normally you will not be looking at colors and your pictures side by side. Because of that, you may assume that an average Delta-E94 of less than six is an excellent result for a modern camera.
Color accuracy (Delta-E94) @ tungsten
  • Average 16,4 (RAW) / 20,4 (jpg)
  • Best 14,8 RAW @ 100 ISO
  • Worst 36,5 jpg @ 25.600 ISO
  • Skin tones 13,1 (RAW) / 21,4 (jpg)
  • Natural colors 17,6 (RAW) / 22 (jpg)
  • Bright colors 13,4 (RAW) / 16,8 (jpg)

Autofocus accuracy

Many photographers rely almost blindly on auto focus. That in itself is understandable, as cameras focus faster then you can focus yourself. We test focus accuracy of a camera with a series of 20 shots. In our procedure, the camera starts focusing from the closest focus and from infinity. We compare the sharpness of these 20 recordings to assess the autofocus accuracy. The accuracy of the autofocus is not only dependent on the camera, but also of the lens and the selected focus point. We always test AF using the central focus point. For a camera test, we pick a lens which in previous tests scored well.

Autofocus Tracking

The continuous tracking of a moving subject while making a series of pictures is one of the tougher AF tests for a camera. We take a series of pictures of a train coming towards the camera. Subsequently we assess the sharpness of the individual shots. We try to keep our tests as constant over time as possible. But, as camera AF will improve, we will adjust this test in the future.

Image Stabilization

To get a sharp picture, the camera must be kept still. The higher the resolution of a camera, or the longer the focal length of a lens, the more important this becomes. Image stabilization helps you keeping the camera steady. Some brands (Canon, Nikon, Panasonic) have image stabilization built into the lenses. Other brands (Olympus, Sony, Pentax) have image stabilization built into the camera body. Effectiveness of stabilization depends on both the camera and the lens you use. We have chosen to discuss image stabilization in the lens reviews, regardless whether in fact the stabilization takes place in the lens or the camera body. This allows you to compare the performance of different cameras and lenses more easily. If you want to know how we test image stabilization, click here .


Ivo Freriks
Author: Ivo Freriks
With Camera Review Stuff I hope to make a modest contribution to the pleasure that you get from photography. By testing cameras and lenses in the same way, evluating the results and weighing up the pros and cons, I hope to help you find the right camera or lens.

I appreciate it if you share this:

Free Bokeh Beauty eBook