Quick, get the popcorn: Amazon Web Services says Microsoft's benchmarks for Azure are a load of stripe

Fight! Fight! Fight!

The cloudy database world was plunged into drama at the close of this week, as Amazon Web Services locked horns with Microsoft in a spat over benchmarks.

In IT, there are lies, damned lies, and benchmarks. Vendors love quietly and sneakily crafting environments and test software that puts their kit in the best possible light against rivals. Intel, for one, turned these shenanigans into a performance art. It's why everyone should dump a truck of salt on any manufacturer-supplied results.

Microsoft has been bandying around some impressive statistics for SQL on Azure of late, and one particular emission, at the start of December, caused the Bezos brigade to spit out their collective dummy. "Faster and cheaper: SQL on Azure continues to outshine AWS," shrieked Redmond's headline.

Oh, no, it doesn't, AWS snapped back yesterday, a mere two months later.

Microsoft made all manner of lofty claims, with careful footnotes, to the effect that Azure was "up to 3.4 times faster and up to 87 per cent less expensive than AWS EC2." Citing data from a Microsoft-sponsored report by blog-turned-analyst-org GigaOm, the Windows giant also alleged its Azure Ultra Disk was able to comfortably spank the capabilities of AWS-provisioned IOPS volumes.

With hands on hips and tightly pursed lips, AWS responded with a Twitteresque "well, actually…" and wheeled out its own benchmarks, using the publicly available TPC-C HammerDB benchmark tool, showing AWS has 1.75x performance advantage over Redmond's effort and up to a 40 per cent price/performance advantage.

When Microsoft dissed the competition last year, we were worried that a benchmark-off might kick off, and sure enough it is time to load up the popcorn machine.

Amazon complained that striping was used to pump up Azure's performance, which wasn't used on AWS. It also whinged that an old-generation AWS instance was used rather than the latest and greatest (and comparable) that Amazon has to offer. A modified TPC-E benchmark – replete with proprietary Microsoft benchmark tooling – was also used, and, finally, the costs were skewed by leaving out licensing and Software Assurance.

Amazon's top brass, memorably, lobbed its toys skywards last year when Microsoft fiddled with its terms to prohibit cloud deployments in AWS, and others, of software with licenses purchased without Software Assurance and mobility rights. It is therefore of little surprise that it should bite back in this instance.

For its part, Microsoft unloaded the caveat wagon beneath its self-sponsored statistics for those brave souls able to decipher the small print. Memorably, it said: "Actual results and prices may vary based on configuration and region."

Unsurprisingly, the Windows behemoth had no comment when The Register asked it for its take on the AWS slapdown.

Both Microsoft and Amazon recorded impressive growth in their respective cloud businesses in their most recent financial results but the latter's commanding lead is suffering erosion as the former continues to gain ground. As such, we fully expect to see more benchmark-based sniping as the gap continues to close and gloves are removed. ®

Speaking of Amazon... The internet titan today claimed it paid more than $1bn in US federal income taxes in 2019, more than $2.4bn in other federal taxes, including payroll taxes and customs duties, and more than $1.6bn in state and local taxes, including payroll taxes, property taxes, state income taxes, and gross receipts taxes, in 2019.

Also, it was served 1,841 subpoenas, 440 search warrants, and 114 other court orders in the final six months of 2019 by investigators seeking user data.