The Chinese manufacturer Huawei has been distributing devices with secret benchmark modes enabled, drastically misrepresenting the performance of these devices and inflating their test results. It’s far from the first time we’ve been down this road, either with benchmarking or mobile devices, but what Huawei has done is a little unusual.
Anandtech stumbled on this issue while testing multiple Huawei devices and confirmed the device behavior by using private versions of the public benchmarks you can download online. These private versions are code-identical to the standard variants, but use obfuscated names or hide other details to prevent a smartphone from detecting they are running and artificially boosting performance.
As Anandtech explains, what Huawei does is a little different from what other SoC vendors have done in the past. When we’ve seen these issues before, it’s typically been because smartphone vendors were ignoring their own thermal guidelines and throttling behavior in specific contexts and running the SoC full-tilt as long as possible to produce better benchmark scores. But Huawei is approaching the concept from a different angle.
During standard operation, the device’s performance is sharply constrained to hold within its thermal envelope. The Kirin 970 never approaches its advertised clocks as a result. The only time the chip performs at its claimed specifications is when its running certain benchmarks. The only way to do that is to blow out the SoC’s TDP. Anandtech measured the SoC drawing an average of 4.39W when tested using benchmarks where the phone couldn’t cheat, versus 8.57W when the device was allowed to detect the benchmark and run at top clock. This performance is completely unsustainable over the long term (if tests are left running, the devices all start displaying overheating notices).
Technically, one can’t argue that Huawei is overclocking their SoC. After all, mobile manufacturers regularly advertise boost clocks as if they were baseline frequencies and they provide no information on when or if users will see those clocks. But the problem here isn’t whether the Kirin 970 runs at any particular frequency — it’s whether the performance end-users see when they run a graphics benchmark has anything to do with the performance they’ll see when they run a game.
The most basic, fundamental test of any benchmark is whether it captures an aspect of device performance or capability that can guide the end-user in evaluating said device. By operating its smartphone in a different mode when these tests are run, Huawei has made it impossible for tests like GFXBench to provide accurate data. It ultimately doesn’t matter whether the SoC is exceeding its published clock speeds or not. What matters, fundamentally, is that the device is operating in a different mode when and only when end users are attempting to gauge its relative performance next to other devices they might buy instead.
According to Huawei, these changes are the result of decisions the company made to rein in power consumption on its devices and improve battery life. Implementing throttling and strict TDP limitations in almost all use cases can improve battery life while having a minimal impact on the user experience, provided the clocks are well-calibrated. Since upgrading to an iPhone SE earlier this spring, I actually keep the device in low power mode almost all the time. Yes, I take a small performance hit — but the phone battery lasts far longer, overall battery health should drop less quickly, and the actual impact of the lowered CPU clock is almost unnoticeable.
The problem with Huawei’s changes is that the company attempted to eat its cake and have it, too. According to Huawei, it took these steps because benchmark cheating in China is simply rampant and the company feels it must engage in some of this behavior as well to compete against other firms. The question of how to compete effectively in an industry when your competition lies regularly about the performance of its own products is a genuine one, and there are no easy answers to that problem. But engaging in the same activity ultimately does not serve the best interests of consumers.
First Intel Hades Canyon, Radeon-GH Benchmarks Leak
Early data from Intel's Core i7-8809G suggest this integrated CPU + GPU packs performance much more like a midrange discrete card.
Futuremark Adopts Parent Company’s Name, Becomes ‘UL Benchmarks’
The benchmarks aren't going away, but "Futuremark" is.
Benchmarking World of Warcraft’s DirectX 12 Support
World of Warcraft added DirectX 12 support last week — we take the new API out for a spin and compare it to DX11.