Huawei Caught Cheating in Smartphone GPU Benchmarks

The Chinese manufacturer Huawei has been distributing devices with secret benchmark modes enabled, drastically misrepresenting the performance of these devices and inflating their test results. It’s far from the first time we’ve been down this road, either with benchmarking or mobile devices, but what Huawei has done is a little unusual.

Anandtech stumbled on this issue while testing multiple Huawei devices and confirmed the device behavior by using private versions of the public benchmarks you can download online. These private versions are code-identical to the standard variants, but use obfuscated names or hide other details to prevent a smartphone from detecting they are running and artificially boosting performance.

As Anandtech explains, what Huawei does is a little different from what other SoC vendors have done in the past. When we’ve seen these issues before, it’s typically been because smartphone vendors were ignoring their own thermal guidelines and throttling behavior in specific contexts and running the SoC full-tilt as long as possible to produce better benchmark scores. But Huawei is approaching the concept from a different angle.

During standard operation, the device’s performance is sharply constrained to hold within its thermal envelope. The Kirin 970 never approaches its advertised clocks as a result. The only time the chip performs at its claimed specifications is when its running certain benchmarks. The only way to do that is to blow out the SoC’s TDP. Anandtech measured the SoC drawing an average of 4.39W when tested using benchmarks where the phone couldn’t cheat, versus 8.57W when the device was allowed to detect the benchmark and run at top clock. This performance is completely unsustainable over the long term (if tests are left running, the devices all start displaying overheating notices).

Technically, one can’t argue that Huawei is overclocking their SoC. After all, mobile manufacturers regularly advertise boost clocks as if they were baseline frequencies and they provide no information on when or if users will see those clocks. But the problem here isn’t whether the Kirin 970 runs at any particular frequency — it’s whether the performance end-users see when they run a graphics benchmark has anything to do with the performance they’ll see when they run a game.

The most basic, fundamental test of any benchmark is whether it captures an aspect of device performance or capability that can guide the end-user in evaluating said device. By operating its smartphone in a different mode when these tests are run, Huawei has made it impossible for tests like GFXBench to provide accurate data. It ultimately doesn’t matter whether the SoC is exceeding its published clock speeds or not. What matters, fundamentally, is that the device is operating in a different mode when and only when end users are attempting to gauge its relative performance next to other devices they might buy instead.

Fighting Cheaters

According to Huawei, these changes are the result of decisions the company made to rein in power consumption on its devices and improve battery life. Implementing throttling and strict TDP limitations in almost all use cases can improve battery life while having a minimal impact on the user experience, provided the clocks are well-calibrated. Since upgrading to an iPhone SE earlier this spring, I actually keep the device in low power mode almost all the time. Yes, I take a small performance hit — but the phone battery lasts far longer, overall battery health should drop less quickly, and the actual impact of the lowered CPU clock is almost unnoticeable.

The problem with Huawei’s changes is that the company attempted to eat its cake and have it, too. According to Huawei, it took these steps because benchmark cheating in China is simply rampant and the company feels it must engage in some of this behavior as well to compete against other firms. The question of how to compete effectively in an industry when your competition lies regularly about the performance of its own products is a genuine one, and there are no easy answers to that problem. But engaging in the same activity ultimately does not serve the best interests of consumers.

Continue reading

Huawei Sells Honor Brand Amid Tightening Trade Restrictions

(Credit: Kevin Frayer/Getty Images)Huawei has been battered by US trade restrictions in the last few years, and it’s taking a toll on the company’s long-term stability. Experts don’t expect a radical change when the new US administration comes to power next year, so Huawei is beginning to take drastic action. It has sold its Honor…

It Turns Out Huawei’s HarmonyOS Is Still Just Android

Following the Commerce Department's actions against the Chinese megafirm, Huawei has been unable to use Google services on its new phones. The company's solution was to develop HarmonyOS, but now that we've gotten our first real look at it, one thing is clear: this is just Android with a skin.

Huawei 2.0: Invading Ukraine Could Cost Russia the Modern Semiconductor Market

The United States is threatening to use export controls to sanction Russia if it invades Ukraine. The Biden Administration would presumably use some of the same tools the Trump Administration previously deployed against Huawei.The Ultimate BanhammerThe United States maintains a list of individuals, corporations, governments, and non-governmental organizations (NGOs) that are subject to export restrictions.…

Canada Bans ZTE and Huawei Network Infrastructure Citing National Security Concerns

Telecom companies in Canada will no longer be able to purchase infrastructure from the companies, and they will have to remove what they have in the coming years.