Futuremark Begins Delisting "Cheating" Mobile Devices from 3DMark Database
by Anand Lal Shimpi on November 26, 2013 9:37 PM EST- Posted in
- Smartphones
- Mobile
- Tablets
Over the summer, following a tip from @AndreiF7, we documented an interesting behavior on the Exynos 5 Octa versions of Samsung's Galaxy S 4. Upon detecting certain benchmarks the device would plug in all cores and increase/remove thermal limits, the latter enabling it to reach higher GPU frequencies than would otherwise be available in normal games. In our investigation we pointed out that other devices appeared to be doing something similar on the CPU front, while avoiding increasing thermal limits. Since then we've been updating a table in our reviews that keeps track of device behavior in various benchmarks.
It turns out there's a core group of benchmarks that seems to always trigger this special performance mode. Among them are AnTuTu and, interestingly enough, Vellamo. Other tests like 3DMark or GFXBench appear to be optimized for, but on a far less frequent basis. As Brian discovered in his review of HTC's One max, the list of optimization/cheating targets seems to grow with subsequent software updates.
In response to OEMs effectively gaming benchmarks, we're finally seeing benchmark vendors take a public stand on all of this. Futuremark is the first to do something about it. Futuremark now flags and delists devices caught cheating from its online benchmark comparison tool. The only devices that are delisted at this point are the HTC One mini, HTC One (One max remains in the score list), Samsung Galaxy Note 3 and Samsung Galaxy Tab 10.1 (2014 Edition). In the case of the Samsung devices, both Exynos 5 Octa and Qualcomm based versions are delisted.
Obviously this does nothing to stop users from running the benchmark, but it does publicly reprimand those guilty of gaming 3DMark scores. Delisted devices are sent to the bottom of the 3DMark Device Channel and the Best Mobile Devices list. I'm personally very pleased to see Futuremark's decision on this and I hope other benchmark vendors follow suit. Honestly I think the best approach would be for the benchmark vendors to toss up a warning splash screen on devices that auto-detect the app and adjust behavior accordingly. That's going to be one of the best routes to end-user education of what's going on.
Ultimately, I'd love to see the device OEMs simply drop the silliness and treat benchmarks like any other application (alternatively, exposing a global toggle for their benchmark/performance mode would be an interesting compromise). We're continuing to put pressure on device makers, but the benchmark vendors doing the same will surely help.
Source: Futuremark
Facebook
Twitter
RSS
46 Comments
View All Comments
OzedStarfish - Wednesday, November 27, 2013 - link
So much butt hurt in those reviews, but at least Futurmark is on the case to explain the situation to those less than happy about their stance, and kudos to them for doing so.Tehk17 - Wednesday, November 27, 2013 - link
Agreeddjscrew - Wednesday, November 27, 2013 - link
It's a shrewd business move and nothing more. Their product is valueless if gamed and knowledge of the game is public, as it has been made by the media.Sunrise089 - Wednesday, November 27, 2013 - link
Huge props to Futuremark.I know AT's public position is that 'name and shame' isn't working, and instead opting for a more measured response of behind the scenes pressure to device makers plus limited public criticism, but I'm pleased Futuremark is standing up and taking the lead on this.
FYoung - Wednesday, November 27, 2013 - link
I don't know how feasible it would be, but I think a preferable approach would be to enhance the software so it could not be cheated, for example by adding code that would randomly change the file name of the benchmark executable each time it is downloaded or run.thunng8 - Wednesday, November 27, 2013 - link
Your suggestion would not work .. Just think about it.hlovatt - Wednesday, November 27, 2013 - link
Great news, hope everyone else including AnandTech follow suit.Creig - Wednesday, November 27, 2013 - link
Good going, Futuremark. It's nice to see them keeping everybody honest. I believe this is only the 2nd time they've called out companies for cheating on their benchmark suite. The first time was Nvidia with 3DMark03. THAT was some over-the-top, blatant cheating, though.http://www.kickassgear.com/downloads/3dmark03_audi...
hrrmph - Wednesday, November 27, 2013 - link
Awesome.Now we need something to go after phone region locking, phone locking (in general), and movie region locking.
tushar001 - Sunday, December 1, 2013 - link
I agree that listing benchmarking apps to provide 100% capacity in delisted device may seems fallacious to Anand, those are benchmarking apps which determine limit of CPU & GPU under rigorous test conditions, so to overpower them is dutiful to smartphones & tabs by provide their best! That's what exactly has HTC & Samsung has done. Considering expectation of benchmarking apps, its primary objective of gadget is expected to run at its maximum capacity to provide accurate readings, so why it is needed to delist a device which specifically runs CPU & GPU at it full clocked speed isn't that limit is tested by benchmarking concept??? Now here it goes fishy that those products have overthrown benchmark's expectation so may be the cause of delisting!!! ya, its hard to digest fact...