DxOMark Adds Selfie Benchmark to Its Phone Scores

Love them or hate them, selfies are here to stay. Phones are by far the most popular cameras today, and selfies are a rapidly growing use of them. So it makes sense that consumers are increasingly choosing their next phone based partially on its ability to capture selfies.

Phone makers have been paying attention. They’ve introduced dual-camera front-facing camera setups for better portrait effects and wide-angle selfie cameras for group selfies. They’ve also bumped up front-facing camera quality and resolution. But until now, there hasn’t been an industry-wide published benchmark for selfie quality. DxOMark Image Labs (most famous for dxomark.com) is aiming to change that with the release of its DxOMark Selfie test suite, along with initial benchmark results for several representative phones.

Rise Of the Selfie

The ability to quickly post to social media added fuel to the fire. The current number of selfies captured is in the billions. By 2014, Google estimated that Android phones alone were used to capture 93 million selfies each day. By 2015, over 25 billion had been uploaded to Google Photos. Samsung estimates the average phone user will take 25,000 selfies over the course of their lifetime.

Selfies Are Harder Than They Look

A panel of models are part of the testing so that skin tones from various races are included

Unfortunately, simply pointing a camera at your face from a foot or two away doesn’t automatically yield a great picture. The cheap front-facing cameras on early smartphones didn’t produce quality images in the first place, and the combination of short distance to the subject and a relatively wide-angle lens distorts faces and bodies in the foreground. Lighting on the subject of the selfie can also be dramatically different from that on the background, making it hard for the camera to pick an exposure that will make the subject look natural and still work for the background.

Ideally, the camera would also know whether to try to blur the background — as most rear cameras now do in portrait mode — or make it as sharp as possible, in the case where the photographer wants to show the subject or group of subjects in a scene.

To address these issues, phone makers have upped their game in both hardware and software. Front-facing cameras are now based on better sensors, often helped out with additional methods of estimating distance. Some companies like Google have opted for wider-angle front cameras to make better group selfies possible, while others have doubled up on their front cameras to provide better Portrait images. HDR mode and multi-frame processing have also migrated from the main to the front camera on many newer phones. Going beyond features typically found in the main camera, most phone makers have also started to add automatic “beautification” capability to their selfie cameras.

With the increasing importance of selfies to both phone owners and phone makers, it makes sense for DxOMark to extend its phone camera test suite to include testing and scoring of a phone’s front-facing camera. Hence, DxOMark Selfie.

This simple comparison shows how differently selfie cameras can capture the same scene. There are variations in color rendering, skin tone, and skin detail between these three phones.

Challenges in Testing Selfie Quality

At its core, measuring image quality for a front-facing camera includes the same basic aspects as for the main, rear-facing, camera. So DxOMark Selfie includes almost the same scoring categories as DxOMark Mobile. For Photo, that’s Exposure, Color, Focus, Texture, Noise, Artifacts, Flash, and Bokeh. Only Zoom is left out. For Video, Exposure, Color, Focus, Texture, Noise, Artifacts, and Stabilization are measured — identical to the categories for main cameras. The testing also follows the same overall structure, including over 1,500 images captured in the lab and outdoors, both tripod-mounted and handheld, along with dozens of sample video clips. But the specifics of testing selfies posed some unique challenges, including short working distances and skin tone variety, that required novel solutions.

DxOMark Selfie includes images in lighting ranging from 1 lux to many thousands of lux.

By definition, selfies include at least one person close to the camera. So for starters, the selfie test protocols focus on the close-range portrait performance of the front-facing camera. DxOMark Selfie includes tests at three distances: 30 cm, 55 cm, and 120 cm, to allow for various ways of holding the phone and the possible use of a selfie stick. You can see the use of this range of distances in these examples:

Selfies captured with subject distances ranging from about a foot out to several feet.

Skin tone rendering is also crucial for selfies, particularly since a light or dark skin tone might require different exposure settings. Group selfies may include a range of skin tones, making the camera’s choice of exposure even more complex. DxOMark has chosen to test phones with the international version of their firmware and default settings. But the company also had to come up with some unique test equipment to make its tests both accurate and repeatable.

The most challenging innovation was the creation of a pair of mannequins, in conjunction with a Hollywood special effects firm, that accurately reproduce the spectral response of various human skin tones, while allowing for repeating test results without the issues associated with trying to have human subjects look, act, and pose identically for multiple day tests.

Benchmarking Beauty

DxOMark Selfie Complements DxOMark Mobile

DxOMark Selfie scores are completely independent of, and can’t directly be compared with, DxOMark Mobile scores for the main camera. Starting now, DxOMark is releasing initial Selfie scores for some phones and will continue to score new models on DxOMark.com, the same way it does front-facing cameras.

Disclosure: The author wrote some of the material DxOMark Image Labs is using to explain and promote DxOMark Selfie.