Google, Microsoft butt heads in browser benchmark battle

Will the real real-world test please stand up?

17 Reg comments Got Tips?

Google Chrome has finally met a web benchmark suite it can't master, and no one could be happier than the Internet Explorer team at Microsoft, which has used the occasion to suggest that the Chocolate Factory's browser is not really the speed demon it's cracked up to be.

Dubbed RoboHornet, the new, open source benchmark suite is different from earlier ones in that it doesn't focus on timing arbitrary JavaScript algorithms. Instead, it collects code snippets found in popular web frameworks and applications – including jQuery, YUI, Google Apps, and others – in an attempt to demonstrate real-world performance problems.

The result is a formidable benchmark suite that takes minutes to complete on typical hardware – assuming the browser can make it through the full list of tests at all.

"It's a living, dynamic benchmark that aims to use the collective efforts of the web development community and ultimately get browser vendors to fix real-world performance pain points," says Google Chrome product manager Alex Komoroske, who launched the project.

That a Google staffer came up with the idea for RoboHornet should surprise no one, since the ad-slinging giant has been responsible for several earlier browser benchmarks. What is surprising, on the other hand, is that Chrome doesn't always earn the highest score, depending on which you run it on.

In tests conducted by Tom's Hardware, Internet Explorer 10 scored as much as 22 per cent higher than the most recent stable version of Chrome when running on Windows 8, and Safari bested Chrome by a similar margin on OS X.

Those findings did not go unnoticed by Microsoft, but despite the strong showing by its latest browser, Redmond was not impressed – not with IE's benchmark scores and not with RoboHornet itself.

"While we appreciate the gesture, members of our engineering team took a look at the benchmark and found that RoboHornet isn't all that representative of the performance users might encounter on real-world sites," wrote Microsoft's Roger Capriotti in a blog post on Tuesday.

To prove it, Microsoft engineers rolled their own version of RoboHornet that they're calling RoboHornet Pro, which Capriotti claims does a better job of modeling what goes on inside the browser when a real-world web app is running.

Redmond's version of the benchmark runs a Matrix-like animation in the foreground while its tests are running in the background. Although Microsoft's truncated test suite takes a lot less time to complete, the animation stutters to a halt on Chrome after a few seconds, while IE10 keeps both the animation and the benchmarks running in tandem.

According to Capriotti, that's because RoboHornet is merely the latest in a long line of browser "micro-benchmarks", which really only demonstrate isolated aspects of browser performance under lab-like conditions.

"While we will look to see how we stack up on other browser vendors' tests, our focus will remain on real-world site performance," he wrote.

Neither Komoroske or any of the other RoboHornet code stewards has commented directly on RoboHornet Pro or Microsoft's findings, but Komoroske at least acknowledges that there is more work to be done.

"RoboHornet is still in a very early alpha state, but that's where you come in," Komoroske writes. "It's up to you and to propose and vote for performance issues you care about, helping shape the future of the benchmark and effectively defining the areas that browser vendors will invest in making run faster!"

Developers who want to get involved can participate via the project's GitHub page. ®

SUBSCRIBE TO OUR WEEKLY TECH NEWSLETTER


Keep Reading

We've come to wish you an unhappy birthday: Microsoft to yank services from Internet Explorer, kill off Legacy Edge by 2021

You need to give that plate back to us after you've finished your cake. Yes the fork too. We'll get your coat

Azure DevOps Services reminds users that, yes, it really is time to pull the plug on Internet Explorer 11

Ignite Sure, it's still wedged in the OS, but maybe you'd prefer something shiny and Chromier?

In a world where up is down, it's heartwarming to know Internet Explorer still tops list of web dev pain points

Incompatibilities and inconsistent standards support among browsers ensure an ongoing source of headaches

Disabled by default: Microsoft ups the ante in its war against VBScript on Internet Explorer

Will the last IE 11 user please turn out the lights?

If you never thought you'd hear a Microsoftie tell you to stop using Internet Explorer, lap it up: 'I beg you, let it retire to great bitbucket in the sky'

We say take off and nuke the entire codebase from orbit. It's the only way to be sure

Bill G on Microsoft's biggest blunder... Was it Bing, Internet Explorer, Vista, the antitrust row?

Nope: It was not giving Android a run for its money...

Microsoft adds Internet Explorer mode to Chromium Edge, announces roadmap

Enterprise features including support for hated ancient browser ready to evaluate

Microsoft will release a web browser for Linux next month. Repeat, Microsoft will release a browser for Linux – and it uses Google's technology

Ignite This means Linus Torvalds has definitely won, doesn't it?

Biting the hand that feeds IT © 1998–2020