Huawei Caught Cheating in Smartphone GPU Benchmarks

The Chinese manufacturer Huawei has been distributing devices with secret benchmark modes enabled, drastically misrepresenting the performance of these devices and inflating their test results. It’s far from the first time we’ve been down this road, either with benchmarking or mobile devices, but what Huawei has done is a little unusual.

Anandtech stumbled on this issue while testing multiple Huawei devices and confirmed the device behavior by using private versions of the public benchmarks you can download online. These private versions are code-identical to the standard variants, but use obfuscated names or hide other details to prevent a smartphone from detecting they are running and artificially boosting performance.

As Anandtech explains, what Huawei does is a little different from what other SoC vendors have done in the past. When we’ve seen these issues before, it’s typically been because smartphone vendors were ignoring their own thermal guidelines and throttling behavior in specific contexts and running the SoC full-tilt as long as possible to produce better benchmark scores. But Huawei is approaching the concept from a different angle.

During standard operation, the device’s performance is sharply constrained to hold within its thermal envelope. The Kirin 970 never approaches its advertised clocks as a result. The only time the chip performs at its claimed specifications is when its running certain benchmarks. The only way to do that is to blow out the SoC’s TDP. Anandtech measured the SoC drawing an average of 4.39W when tested using benchmarks where the phone couldn’t cheat, versus 8.57W when the device was allowed to detect the benchmark and run at top clock. This performance is completely unsustainable over the long term (if tests are left running, the devices all start displaying overheating notices).

Technically, one can’t argue that Huawei is overclocking their SoC. After all, mobile manufacturers regularly advertise boost clocks as if they were baseline frequencies and they provide no information on when or if users will see those clocks. But the problem here isn’t whether the Kirin 970 runs at any particular frequency — it’s whether the performance end-users see when they run a graphics benchmark has anything to do with the performance they’ll see when they run a game.

The most basic, fundamental test of any benchmark is whether it captures an aspect of device performance or capability that can guide the end-user in evaluating said device. By operating its smartphone in a different mode when these tests are run, Huawei has made it impossible for tests like GFXBench to provide accurate data. It ultimately doesn’t matter whether the SoC is exceeding its published clock speeds or not. What matters, fundamentally, is that the device is operating in a different mode when and only when end users are attempting to gauge its relative performance next to other devices they might buy instead.

Fighting Cheaters

According to Huawei, these changes are the result of decisions the company made to rein in power consumption on its devices and improve battery life. Implementing throttling and strict TDP limitations in almost all use cases can improve battery life while having a minimal impact on the user experience, provided the clocks are well-calibrated. Since upgrading to an iPhone SE earlier this spring, I actually keep the device in low power mode almost all the time. Yes, I take a small performance hit — but the phone battery lasts far longer, overall battery health should drop less quickly, and the actual impact of the lowered CPU clock is almost unnoticeable.

The problem with Huawei’s changes is that the company attempted to eat its cake and have it, too. According to Huawei, it took these steps because benchmark cheating in China is simply rampant and the company feels it must engage in some of this behavior as well to compete against other firms. The question of how to compete effectively in an industry when your competition lies regularly about the performance of its own products is a genuine one, and there are no easy answers to that problem. But engaging in the same activity ultimately does not serve the best interests of consumers.

Continue reading

In Leaked Memo, LG Proposes Withdrawing From Smartphone Market

A memo circulated at LG declares some hard choices are necessary after $4.5 billion in losses over the past five years. The memo lists several possible courses of action, including the end of LG's mobile business.

Sony’s New Xperia Pro Smartphone Is a $2,500 Camera Accessory

This device is aimed at professional and enthusiast photographers who want a powerful accessory for their cameras. It's also Sony's first 5G smartphone in the US, but the price is positively jaw-dropping at $2,500.

A 15-Minute Smartphone COVID Test Is Coming to the US

The Department of Defence and Department of Health and Human Services have awarded $230 million to an Australian biotech firm Ellume to produce a 15-minute COVID test that sends results directly to your phone.

Report: LG Can’t Find Anyone to Buy Its Smartphone Business

LG has been trying to turn its mobile unit around for the last several years, but it has yet to pull out of the red. In early 2021, rumors suggested LG was looking to sell off its smartphone business. Now, a new South Korean report says LG has tried and failed to find a suitor, and the next step might just be to shut down the division entirely.