Apple accused of cheating over G5 benchmarks

SPEC vs SPEC


Benchmark results cited by Apple at the launch of its Power Mac G5 desktops yesterday have already come under fire for seeming to not only tweak the Mac test system to improve its performance beyond anything an ordinary user might experience, but crippling rival systems to deliver below-par average user performance.

The tests described by Apple CEO Steve Jobs were conducted on the company's behalf ("under contract") by VeriTest. The benchmarks used are SPEC CPU 2000 integer and floating-point tests. Apple asked VeriTest to compare a pre-release a dual 2GHz Power Mac G5 with a Dell Precision 650 workstation based on twin 3.06GHz Intel Xeon CPUs and a Dell Dimension 8300 based on a 3GHz Pentium 4.

The Dell's were running Red Hat Linux 9.0, the G5 Mac OS X 10.2.7. The test software was compiled using GCC 3.3 and NAGware Fortran 95.

VeriTest recorded SPECint base score of 800, 889 and 836 for the G5, 8300 and 650, respectively. The equivalent SPECfp base scores were 840, 693 and 646. So the G5 out-performs the other machines, yes?

Well, so says Apple, but a closer look at VeriTest's documentation, freely available from its web site, suggests otherwise.

Certainly SPEC figures published on the SPEC web site do, as Register readers noted, along with readers at a number of web sites today. The corresponding SPECint and SPECfp base Dell-provided results for the 650 are 1089 and 1053. Equivalent figures for the Dimension 8300 are not available.

That puts Apple's figures in a new light. On one hand we have figures that suggest the 2GHz G5 outperforms the 3GHz Xeon in certain benchtests, and on the other we have numbers that show the exact opposite. What gives?

Firstly, Dell's own figures were calculated using different compilers and host operating system: Windows XP Pro, Intel's own C++ and Fortran compilers, and the MicroQuill SmartHeap Library 6.01. Secondly, the compiler used by VeriTest, GCC, is said to generate code that less well optimised for x86. Thirdly, VeriTest seems to have adjusted the test hardware to favour the G5. Again, all the details are there in the documentation.

VeriTest admits it used an Apple-supplied tool to adjust the G5 processor's registers "to enable Memory Read Bypass" and "to enable the maximum of eight hardware prefetch streams and disable software-based pre-fetching". The company also installed a "high performance, single-threaded malloc library... geared for speed rather than memory efficiency". That, says VeriTest, "makes it unsuitable for many uses".

We'd guess these are hardly standard system configurations.

VeriTest also says it tweaked the Dell boxes. For example, when it came to the SPECint and SPECfp rate tests, it disabled HyperThreading, though enabled it for the base SPECint and SPECfp tests. While the compilers were set to optimise code for the Pentium 4, SSE 2 instructions were not used to speed floating point maths operations, only SSE 1 instructions were enabled. VeriTest provides no clear rationale for these choices.

Without that rationale, both VeriTest and Apple are today widely being accused of cheating, and not only by x86 fans eager to see the G5 knocked a peg or two back down the CPU performance ladder. To be fair, at least Apple and VeriTest tell you what they've done, which is more than can be said for the vendor-supplied figures on SPEC's web site. What tweaks have vendors applied to boost their own scores? But they should also say why.

The VeriTest test appears to be an attempt to get the apples/oranges comparison as close as they possibly can, but looking deeper suggests that they have failed to do so.

As we noted in our story on Apple's G5 introduction, we await independent, real world tests of the new Power Mac G5's performance. Only then will Mac users - and everyone else, for that matter - get a truly worthwhile comparison between platforms. ®


Other stories you might like

  • How ICE became a $2.8b domestic surveillance agency
    Your US tax dollars at work

    The US Immigration and Customs Enforcement (ICE) agency has spent about $2.8 billion over the past 14 years on a massive surveillance "dragnet" that uses big data and facial-recognition technology to secretly spy on most Americans, according to a report from Georgetown Law's Center on Privacy and Technology.

    The research took two years and included "hundreds" of Freedom of Information Act requests, along with reviews of ICE's contracting and procurement records. It details how ICE surveillance spending jumped from about $71 million annually in 2008 to about $388 million per year as of 2021. The network it has purchased with this $2.8 billion means that "ICE now operates as a domestic surveillance agency" and its methods cross "legal and ethical lines," the report concludes.

    ICE did not respond to The Register's request for comment.

    Continue reading
  • Fully automated AI networks less than 5 years away, reckons Juniper CEO
    You robot kids, get off my LAN

    AI will completely automate the network within five years, Juniper CEO Rami Rahim boasted during the company’s Global Summit this week.

    “I truly believe that just as there is this need today for a self-driving automobile, the future is around a self-driving network where humans literally have to do nothing,” he said. “It's probably weird for people to hear the CEO of a networking company say that… but that's exactly what we should be wishing for.”

    Rahim believes AI-driven automation is the latest phase in computer networking’s evolution, which began with the rise of TCP/IP and the internet, was accelerated by faster and more efficient silicon, and then made manageable by advances in software.

    Continue reading
  • Pictured: Sagittarius A*, the supermassive black hole at the center of the Milky Way
    We speak to scientists involved in historic first snap – and no, this isn't the M87*

    Astronomers have captured a clear image of the gigantic supermassive black hole at the center of our galaxy for the first time.

    Sagittarius A*, or Sgr A* for short, is 27,000 light-years from Earth. Scientists knew for a while there was a mysterious object in the constellation of Sagittarius emitting strong radio waves, though it wasn't really discovered until the 1970s. Although astronomers managed to characterize some of the object's properties, experts weren't quite sure what exactly they were looking at.

    Years later, in 2020, the Nobel Prize in physics was awarded to a pair of scientists, who mathematically proved the object must be a supermassive black hole. Now, their work has been experimentally verified in the form of the first-ever snap of Sgr A*, captured by more than 300 researchers working across 80 institutions in the Event Horizon Telescope Collaboration. 

    Continue reading
  • Shopping for malware: $260 gets you a password stealer. $90 for a crypto-miner...
    We take a look at low, low subscription prices – not that we want to give anyone any ideas

    A Tor-hidden website dubbed the Eternity Project is offering a toolkit of malware, including ransomware, worms, and – coming soon – distributed denial-of-service programs, at low prices.

    According to researchers at cyber-intelligence outfit Cyble, the Eternity site's operators also have a channel on Telegram, where they provide videos detailing features and functions of the Windows malware. Once bought, it's up to the buyer how victims' computers are infected; we'll leave that to your imagination.

    The Telegram channel has about 500 subscribers, Team Cyble documented this week. Once someone decides to purchase of one or more of Eternity's malware components, they have the option to customize the final binary executable for whatever crimes they want to commit.

    Continue reading
  • Ukrainian crook jailed in US for selling thousands of stolen login credentials
    Touting info on 6,700 compromised systems will get you four years behind bars

    A Ukrainian man has been sentenced to four years in a US federal prison for selling on a dark-web marketplace stolen login credentials for more than 6,700 compromised servers.

    Glib Oleksandr Ivanov-Tolpintsev, 28, was arrested by Polish authorities in Korczowa, Poland, on October 3, 2020, and extradited to America. He pleaded guilty on February 22, and was sentenced on Thursday in a Florida federal district court. The court also ordered Ivanov-Tolpintsev, of Chernivtsi, Ukraine, to forfeit his ill-gotten gains of $82,648 from the credential theft scheme.

    The prosecution's documents [PDF] detail an unnamed, dark-web marketplace on which usernames and passwords along with personal data, including more than 330,000 dates of birth and social security numbers belonging to US residents, were bought and sold illegally.

    Continue reading

Biting the hand that feeds IT © 1998–2022