The wonderful madness of metrics: Different things to different folk

Or, how I learned to stop worrying and verify


Managers and customers love statistics and metrics. Companies can live or die by how good their metrics are and the potential penalties for failing to meet the required service levels as defined in agreements.

It can also be: “Have my team met their SLA” or: “What is the uptime on the server farm”.

The dictionary defines the noun metric as: “A standard for measuring or evaluating something, especially one that uses figures or statistics.”

In an honest world, things would be that simple. But this isn’t an honest world. Metrics can be interpreted differently, stretched, used and abused if they are not laid out in black and white. It can be like comparing apples and oranges, to use an often quoted phrase.

It goes without saying that any technology vendor has an interest in metrics. Metrics provide a way of measuring performance — remember that!

Obviously the better the score for the product being tested, the better the vendor in question looks to its customers. To be kind we could say that metrics (essentially test methodologies) are sometimes open to interpretation and vendor interpretation of the methodology used for a supposedly standard test.

As positive metrics mean good money it should come as no surprise that some vendors have been caught tweaking metrics, software or hardware to make their product look good. Many vendors make claims using “optimised” metrics that are skewed to represent their product in a good light.

Currently there are a lot of examples in the smart device arena where manufacturers have actually come clean about being caught. Some vendors had a special “secret mode” that when a test platform was detected, it went all out to get good scores, heat and battery usage be damned.

Yet another frequent example is storage vendors claiming massive IOPS (I/O Operations Per Second) from their array. Yes, a certain array may well perform that many IOPS but only under very controlled conditions and specific configurations.

Not all IOPs are created equal. Large sequential reads require fewer IOPs than a random write operation.

Real-world IOPS are a whole lot less as it isn’t as non random as the test the manufacturer performed. Also the machine evaluated won’t be the entry level unit for sure! As a side note on the storage side, things have improved in the storage testing methodology arena but there is definitely room for improvement.

Another form of metrics that often causes a lot of contention is around website uptimes and monitoring. A lot of websites an X-as-a-Service advertise an uptime of 99.999 per cent. To those who can’t be bothered doing the maths, five nines equates to 5.26 minutes per year or 25.9 seconds of permissible downtime per month.

That in itself is a tall order for even the biggest of vendors/data centres with bags of cash and staff.

If that’s the case, how are all these smaller vendors claiming to be anywhere near five nines. The truth is, the devil is in the detail (otherwise known as the small print). Frequently contracts that get signed for leased network lines, cloud, websites and such, have clauses that will commonly include the following exclusions:

  1. Act of God — so any hurricanes or a plane flying into your data center or such doesn’t count
  2. Issues beyond vendor control — such as the network lines get torched, as happened recently in central London
  3. Patching and updates for security requirements
  4. Planned downtime

Other stories you might like

  • NASA installs a new and improved algorithm to better track near-Earth asteroids

    Nearly 20 year-old software used to protect humanity gets an upgrade

    NASA has upgraded its near-Earth asteroid monitoring algorithm to model hazardous space rocks more accurately after nearly two decades, it announced on Tuesday.

    The new system, dubbed Sentry-II, is more powerful than its predecessor, Sentry. Astronomers working at the space agency's Center for Near Earth Object Studies can now automatically calculate thermal influences that nudge an asteroid’s orbit, potentially sending it hurtling towards our home planet.

    The so-called Yarkovsky effect describes the subtle and gradual change of motion when asteroids are heated by the Sun’s light. When asteroids spin, one side of its surface exposed to the star gets heated. As it continues to rotate, the hot region enters shade and cools down. Infrared energy is radiated outwards; the photons carry momentum and impart a tiny thrust on the asteroid. Over long periods of time, these small kicks can change their paths and knock them out of their original orbit.

    Continue reading
  • Facebook slapped with an eyepopping $150B lawsuit for spreading hate speech against Rohingya refugees

    Lawsuit claims social media giant's algos helped Myanmar military crackdown on the Rohingya

    Meta was sued on Tuesday for a whopping $150 billion in a class-action lawsuit for allegedly amplifying hate speech and aiding the Myanmar military in the genocide of the Rohingya people.

    The case, led by an anonymous Rohingya refugee living in the US, accuses the entity formerly known as Facebook of inciting hatred and inflicting real harm on the predominantly Muslim group for years. Not only did the social media platform ignore hate speech posts, it's alleged that the service's algorithms actively promoted anti-Rohingya propaganda as hundreds of thousands of people fled from Myanmar to escape persecution.

    Facebook has already acknowledged its role in the campaign, which saw an estimated 25,000 people perish and 700,000 forced from the country. The lawsuit also comes after ex-employee and whistleblower Frances Haugen leaked internal documents demonstrating how its algorithms prioritized engagement over safety.

    Continue reading
  • Power management IC shortage holding cars, laptops, hostage

    Couple of cents-worth of kit causing big problems for the year to come

    The shortage of power management chips is worsening and holding back companies from building cars, PCs and items with batteries or an on-off switch, Trendforce said in a study this week.

    Power management ICs cost just a few cents, and are among cheap chips that include display driver and USB-C components that are in short supply. These chips are as important to PCs and other electronics as CPUs or memory.

    The demand for PMICs has gone through the roof with the emergence of electric cars and growing demand for PCs and consumer electronics during the past 20 plus months. Trendforce expects the prices will go up by 10 per cent to a six-year high of $0.23.

    Continue reading

Biting the hand that feeds IT © 1998–2021