"That's the highest score of any smartphone camera."

Over the years that statement has been waved at consumers at all major product releases. The DxOMark score has become a staple in these launches and a crucial benchmark for manufacturers.

The score features in article headlines, Twitter gets flooded with DxOMark tweets by CEOs, and manufacturers wear it like a badge of honor. But there's a problem with trying to use a single number to encompass a camera's quality -- it's somewhat accurate, but at the same time can be very misleading.

Why DxOMark?

DxOMark has become the yardstick for digital cameras, lenses, and smartphone cameras. This approach of leveling the playing field is crucial in comparing one camera to another. DxOMark's testing protocols follow clinical-style procedures including laboratory conditions, methods, and test-specific tools to ensure repeatable results. When DxOMark releases a score, that score should reflect an unbiased, scientific, objective number that sets a benchmark for manufacturers to compete against.

That's the appeal of the DxOMark score. But does it really hold up?

The Testing Methodology

All smartphones are judged based on their cameras in Auto mode, and the results are extracted from the JPEG files they produce. Manual adjustments are only made when testing bokeh, zoom, and flash capabilities.

There's a rigorous set of laboratory-style tests followed by "50 realistic and challenging indoor and outdoor scenes." DxOMark claims that these results are reviewed by a "panel of imaging experts."

Above is one of the reference sets that DxOMark uses in one of its tests. Once an image is produced, the panel of imaging experts will use an image quality ruler like the one seen below.

Looking at the new image taken by the smartphone, they will match it to the closest one on the image quality ruler. This is how they turn a qualitative measurement into the quantitative value that gets plastered on billboards.

The Scoring

A number of sub-scores make up the number that you see in the headlines. Tests are first divided into photo and video. There are then individual scores for:

• Exposure and contrast
• Color
• Autofocus
• Texture
• Noise
• Artifacts
• Flash
• Zoom
• Bokeh

To find these individual scores for a camera you need to look past the headline number. Go to the DxOMark site, find the review for the device you're looking for, and scroll through the review all the way to the bottom.

Taking a look at the Pixel 2 scores below, it is clearly not an average of all the combined scores. Scoring seems to be weighted in specific areas and almost ignored in others. The problem here is that while DxO Labs has tried to remain objective in their approach, you could argue that their weighting system is entirely subjective.

Buying Based on Score

When choosing a camera to buy, there are a lot of questions that need asking. Do you take a lot of low-light photos? Are you going to use bokeh extensively? How often do you use zoom? This is where the overall score can be misleading.

Let's say you need a smartphone that has ample zoom capabilities. Do you look at the leaderboard and pick the one at the top? Right now, that's the Pixel 2.

According to DxO Labs, however, you will be better off with a Note 8, which has a lesser overall score but more than double the zoom score of the Pixel 2.

Another consideration is the fact the DxO Labs will work with manufacturers, for a fee, to better their DxOMark score. This blurs the line slightly for two reasons.

First, if a manufacturer is tuning their camera to perform better in a benchmark, does that mean the same tweaks are better for the consumer?

Second, if a company does everything it can to make an excellent camera but it cannot afford to work with DxO Labs, does that mean they will get a lower score? It might mean DxO won't even test their camera.

Working with DxO Labs might result in a better overall camera, but this weird relationship which leads to better scores means the scores should be taken with a grain of salt.

Close to Perfection?

As technology advances, camera sensors, lenses, resolutions, and frames per second will get better. But the best cameras of today already score 98, so what about next year? Will the 98 mean something different in the future? What about the RED Helium 8KS35 that scored 108? Is it better than perfect?

The mistake that many tech websites and channels make is thinking the DxOMark score is out of a 100. According to DxO Labs, this is just not the case. In fact, they are confident smartphones will surpass the 100 score number.

The fact that a smartphone scored 98 does not mean it's near perfect. In fact, it's pure coincidence. It may look like the camera's score is near perfect, but that's deceiving.

This is not going to stop tech news sites making a big deal when the first smartphones scores 100. Nor is it going to stop them when the first smartphone surpasses 100. But remember: the overall score is always going to increase as cameras get better.

Should I Believe the DxOMark Score?

Yes. They won't compromise being the reference for image quality via poor test methods or dubious practices. DxO Labs does perform solid testing. The issue is primarily with the way the scores are being propagated by the press.

Touting the new highest overall score trying to get clicks, without actually taking a closer look at the DxO review, is misleading.

When you make a purchase, take the time to look past the overall score and look deeper into the DxOMark review. The next time you see a smartphone making a headline for the highest DxOMark score ever, understand that this is to be expected. You should still look deeper into the results.

And remember that there might be some stellar cameras out there that just didn't get the full DxOMark treatment. They're easy to overlook.

What considerations do you make when making a purchase? Have DxOMark scores affected your buying process? What other benchmarks do you consider important? Share with us in the comments below!