Huawei Caught Cheating in Smartphone GPU Benchmarks
The Chinese manufacturer Huawei has been distributing devices with secret benchmark modes enabled, drastically misrepresenting the performance of these devices and inflating their test results. It’s far from the first time we’ve been down this road, either with benchmarking or mobile devices, but what Huawei has done is a little unusual.
Anandtech stumbled on this issue while testing multiple Huawei devices and confirmed the device behavior by using private versions of the public benchmarks you can download online. These private versions are code-identical to the standard variants, but use obfuscated names or hide other details to prevent a smartphone from detecting they are running and artificially boosting performance.
As Anandtech explains, what Huawei does is a little different from what other SoC vendors have done in the past. When we’ve seen these issues before, it’s typically been because smartphone vendors were ignoring their own thermal guidelines and throttling behavior in specific contexts and running the SoC full-tilt as long as possible to produce better benchmark scores. But Huawei is approaching the concept from a different angle.
During standard operation, the device’s performance is sharply constrained to hold within its thermal envelope. The Kirin 970 never approaches its advertised clocks as a result. The only time the chip performs at its claimed specifications is when its running certain benchmarks. The only way to do that is to blow out the SoC’s TDP. Anandtech measured the SoC drawing an average of 4.39W when tested using benchmarks where the phone couldn’t cheat, versus 8.57W when the device was allowed to detect the benchmark and run at top clock. This performance is completely unsustainable over the long term (if tests are left running, the devices all start displaying overheating notices).
Technically, one can’t argue that Huawei is overclocking their SoC. After all, mobile manufacturers regularly advertise boost clocks as if they were baseline frequencies and they provide no information on when or if users will see those clocks. But the problem here isn’t whether the Kirin 970 runs at any particular frequency — it’s whether the performance end-users see when they run a graphics benchmark has anything to do with the performance they’ll see when they run a game.
The most basic, fundamental test of any benchmark is whether it captures an aspect of device performance or capability that can guide the end-user in evaluating said device. By operating its smartphone in a different mode when these tests are run, Huawei has made it impossible for tests like GFXBench to provide accurate data. It ultimately doesn’t matter whether the SoC is exceeding its published clock speeds or not. What matters, fundamentally, is that the device is operating in a different mode when and only when end users are attempting to gauge its relative performance next to other devices they might buy instead.
Fighting Cheaters
According to Huawei, these changes are the result of decisions the company made to rein in power consumption on its devices and improve battery life. Implementing throttling and strict TDP limitations in almost all use cases can improve battery life while having a minimal impact on the user experience, provided the clocks are well-calibrated. Since upgrading to an iPhone SE earlier this spring, I actually keep the device in low power mode almost all the time. Yes, I take a small performance hit — but the phone battery lasts far longer, overall battery health should drop less quickly, and the actual impact of the lowered CPU clock is almost unnoticeable.
The problem with Huawei’s changes is that the company attempted to eat its cake and have it, too. According to Huawei, it took these steps because benchmark cheating in China is simply rampant and the company feels it must engage in some of this behavior as well to compete against other firms. The question of how to compete effectively in an industry when your competition lies regularly about the performance of its own products is a genuine one, and there are no easy answers to that problem. But engaging in the same activity ultimately does not serve the best interests of consumers.
Continue reading
OnePlus Caught Manipulating Smartphone Performance Benchmarks
Despite having ample power, the OnePlus 9 is aggressively throttling performance for most apps while allowing benchmarks full power. As a result, its phones have been pulled from the Geekbench charts.
Western Digital Caught Bait-and-Switching Customers With Slow SSDs
Western Digital is the latest company caught bait-and-switching its customers with lousy SSD speeds. In other news, Samsung and Intel don't appear to be trying to cheat customers this way, while Crucial and Adata apparently are.
Samsung Is the Latest SSD Manufacturer Caught Cheating Its Customers
Samsung has joined the ranks of companies found shipping faster SSDs to reviewers than to consumers without telling anyone about it.
Family Safety App Life360 Caught Selling User Location Data
Keeping track of your family now comes with an unexpected price: handing over your location data to Amazon and the US Military.