Benchmark apps have been existing for a long time now and almost every smartphone manufacturer is boasting the high benchmark scores that their smartphones achieve in these apps. While benchmark scores need not necessarily translate into real-time performance, smartphone companies in the past have been accused of manipulating the scores. On similar lines now, MediaTek has been accused of cheating mobile benchmark and providing it as a service.
According to a report from AnandTech, in the PCMark benchmark, MediaTek Helio P95 SoC on the OPPO Reno3 Pro outperformed the MediaTek Dimensity 1000L SoC on the OPPO Reno3 (Chinese variant). When the benchmark was run again on an anonymized version of PCMark, there was a 30% difference compared to the regular PCMark app installed on these devices, and it isn’t a small difference. In order to check if the issue persisted in Snapdragon-powered devices, the benchmark was run again on the Snapdragon 765G powered OPPO Reno3 Pro 5G, and there was no abnormality found in benchmark scores and it was seemingly similar across all benchmark apps.
Upon further inspection of the firmware files, AnandTech discovered a “Sports Mode” in the APK ID of PCMark. This “Sports Mode” was also found on various other benchmarking apps including AnTuTu, GeekBench, 3D Bench, and other AI Benchmark apps. Essentially, the characteristics of the Dynamic voltage and frequency scaling (DVFS) is fixed on this “Sports Mode” that adjusts the speed and power settings of the device’s processor, and the memory controller can be run at maximum frequency all the time. The firmware files also include “_FPS_ entries” which is also believed to boost the benchmark numbers.
Furthermore, the findings reveal that several devices with MediaTek chipsets had a similar config file with the “Sports Mode”, even on older devices like the Sony XA1 which is powered by the MediaTek Helio P20 SoC.
In response to the report, MediaTek said that it follows industry standards and is confident that benchmarking tests accurately represent the capabilities of its chipsets. “Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements”, says MediaTek. It also says that it is left to the brands to configure their own devices as they see fit.
Furthermore, the company says that benchmarking tests are “in line with the practices of other companies” and also accuses “key competitor” (probably Qualcomm) having chipsets that operate in the “exact same way”.
We do find it interesting that AnandTech has called into question the benchmarking optimizations on MediaTek powered devices, when these types of configurations are widely practiced across the industry. If they were to review other devices, they would see, as we have, that our key competitor has chipsets that operate in the exact same way – what AnandTech has deemed cheating on device benchmarking tests.
MediaTek’s statement to AnandTech:
MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.
Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.
We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.