UL Delists Huawei Benchmark Scores After Cheating Discovered
UL, which owns 3DMark, has issued its own statement on Huawei’s benchmarking practices after the company was discovered to be artificially inflating its benchmark performance by pushing its SoCs out of thermal bounds when benchmarks were added to a whitelist that allowed the SoC to run at clocks that actually match those listed on a phone’s packaging. In all other cases, devices run far below their listed clocks to stay within thermal limits.
Huawei may have come up with this idea as a means to avoid being accused of overclocking its hardware — you can’t argue something is overclocked if it’s only hitting its specified frequencies — but the larger problem is that the company’s hardware tweaks completely blow the power budget and only apply to whitelisted benchmarks.
Huawei and UL have had several conversations on this topic in which Huawei attempted to explain that this wasn’t cheating, but reflected new AI optimizations being used to ensure maximum performance. At the same time, however, these AI optimizations are only being used when the handset is benchmarked in specific tests as opposed to in games — and they’re pushing the devices so far out of bounds, they’re overheating in some cases. Since the entire point of a benchmark is to capture useful information about the performance of a device, a benchmark mode that only runs when the device is being tested is the exact opposite of helpful.
UL writes:
UL understands the intent of Huawei’s approach, but is opposed to forcing the use of a “Performance Mode” by default when a benchmarking application is detected by the device. UL rules require a device to run the benchmark as if it were any other application.
Huawei respects consumers’ right to choose what to do with their devices. Therefore, Huawei will provide users with open access to “Performance Mode” in EMUI 9.0, so that the user can choose when to use the maximum power of their device…
To prevent confusion around current benchmarking results, after discussion, UL and Huawei have temporarily delisted the benchmark scores of a range of Huawei devices, and will reinstate them after Huawei grants all users of Huawei handsets access to the Performance Mode.
This kind of problem is far from new in mobile benchmarking, and while there’s evidence that Huawei is indeed trying to develop a new approach to improving its GPU performance through the use of AI, what the company did here appears far more like a standard whitelist trick to boost performance in the apps people use to test phones by blowing the SoC’s power curve. The device can’t practically operate for any length of time while drawing 8W of power, and that’s what matters — far more than any question of AI optimization or feature usefulness.
Continue reading
Samsung Is the Latest SSD Manufacturer Caught Cheating Its Customers
Samsung has joined the ranks of companies found shipping faster SSDs to reviewers than to consumers without telling anyone about it.
Samsung Caught Cheating Customers Again, But On TVs This Time
Samsung has been caught cheating again, but this time it's on TVs, not SSDs or smartphones.
Dell Accidentally Recommends an 8th Gen Intel CPU When Cheating in PUBG
Dell's latest gaming message was a bit of a muddle.
Bosch: We Can Do Ultra-Low-Emissions Diesels (Without Cheating)
High temperatures (almost 400 degrees) and optimized exhaust gas flow allow diesel exhaust to be 10 times cleaner than today. European diesels should be well under tough 2020 limits.