Measure for measure: Why network surveys don't count what counts

Punters want reliable calls most


Interview If you can't measure something, does it actually exist?

That's a question raised by Dr Paul Carter, founder of the oldest mobile network performance operation, GWS. Carter's company has been doing labour-intensive drive-testing of the networks for over 20 years, which is not cheap, but it is thorough. The more rigorous, scientific approach means that a measurement of call quality and dropped calls can be calculated. But in recent years, a wave of "crowd-sourced" rivals have sprung up, attracted to the market because gathering data on Android phones is cheap.

Crowd-derived data can be fascinating – the ping rates and device performance Tutela gleaned here is genuinely useful. But the problem is, Carter argues, they're not measuring what actually matters most to people.

"The most important network function people want in the UK – and we gave them a long list – is making voice calls. 69 per cent of people want that. Texting is important for 53 per cent of people, and the web third, with 43 per cent. Video streaming was cited as important by just 3 per cent of punters," Carter says.

"Clearly data services are not unimportant – but the differentiator on smartphones is to make voice calls. That's still fundamentally the most important functionality of the phone."

Something else not measured by crowd performance is reliability.

"People want a reliable network more than they want a faster network," Carter adds. Combine these two areas where important stuff isn't being measured, Carter argues, and purely crowd-sourced analysis from operations like Tutela and OpenSignal falls short. Crowd-sourced data cannot test call reliability so it concentrates on measuring what it can: speed.

GWS Top UK cities reliability

GWS's top UK cities for reliability

Rather than whingeing about the upstarts, the Shakespeare-loving founder (Carter is a Trustee of the Royal Shakespeare Company) went back to the drawing board. GWS's response is to include more subjective consumer data – drawn from interviews with punters – into the final OneScore ranking.

In the first survey GWS published using this ranking, there are some surprising results. EE routinely tops the crowd results because it has the fastest network – thanks to scale and spectrum – but out of 32 British towns and cities, GWS finds it's the the most reliable in only three: Leeds, Leicester and Manchester. Hutchison's Three typically ties for last when it comes to speed (usually with O2) – because it held on to 3G HDSPA+ for so long, and has the poorest spectrum allocation – but for reliability it comes top in 14 metro areas, including London and Birmingham. O2 also figures well, being the most reliable network in 17 cities including Liverpool and Glasgow.

How come its so different to crowd-derived results?

"They're looking at throughput and missing the important stuff. They're not even measuring voice calls," Carter says. "Often they're simply doing a capacity test, using multiple threads, to see what the maximum throughput is at that location. In a way that's a non-typical use case. If you were skiing there that's not something you would actually do." If a network pushes newer devices at people then it will get better results, he points out, as older phones won't support newer technologies and frequency bands.

Many of the speed tests are pointless, he argues, once a network reaches a given level. He compares it to the UK's Universities and Colleges Admissions Service.

"If I give you a target of three Bs [at A level] to get in, and you get three As, you're still in – you're not in faster, or more 'in'. Our networks are a little bit like that," he says. "A Facebook or Instagram post could arguably go half a second faster but if you look at a stopwatch, it's hard to save a tenth of a second." Neither is much use if the network isn't reliable.

Carter says GWS isn't out to downplay the importance of data, but to champion the consumer's interest.

"If people could give the networks advice it would be to make them more reliable countrywide, blanket coverage: do everything everywhere; maybe go old school and improve voice with fewer blank spots. It's the same message, over and over again."

O2 also finds itself doing well in the subjective "very satisfied" rating from punters. London suffers worst for dropped calls.

The data was gathered using the Rohde and Schwarz Freerider app – like this one – in late 2017 and early 2018, while the polling and focus groups covered almost 3,000 punters across the UK.

Tutela and OpenSignal are two recent rivals who rely primarily on crowd data. IHS Markit's RootMetrics started using crowd data but switched much more to using drive-testing. The consumers' association Which? attracted criticism for relying on crowd data for its "worse than Peru" report – this year we learned Britain is in worse nick than Armenia.

Carter points out that 81 per cent of punters report they're generally happy with their operator and the satisfaction rating is steadily improving – partly, he suggests, because VoWi-Fi capability is filling in the notspots

"If Donald Trump had an 81 per cent approval rating, he'd be doing backflips." ®

Similar topics


Other stories you might like

  • UK science suffers as lawmakers continue to dither over Brexit negotiations

    Horizons Europe carrot dangled amid protocol wrangling

    A report from the UK House of Commons' European Scrutiny Committee has blamed delays in Brussels for choking off revenue streams to British institutions and businesses.

    The UK departed the European Union following a 2016 referendum. One of the results was that UK businesses were no longer able to tender for lucrative contracts within the bloc.

    The Brexit Divorce Bill uncomfortably laid out the facts back in 2018. The satellite navigation system Galileo was one victim despite substantial involvement from the UK in its development. Another was the Copernicus Earth monitoring programme; the UK was infamously snubbed when the European Space Agency (ESA) handed out six juicy contracts to institutions from the Continent.

    Continue reading
  • Warehouse belonging to Chinese payment terminal manufacturer raided by FBI

    PAX Technology devices allegedly infected with malware

    US feds were spotted raiding a warehouse belonging to Chinese payment terminal manufacturer PAX Technology in Jacksonville, Florida, on Tuesday, with speculation abounding that the machines contained preinstalled malware.

    PAX Technology is headquartered in Shenzhen, China, and is one of the largest electronic payment providers in the world. It operates around 60 million point-of-sale (PoS) payment terminals in more than 120 countries.

    Local Jacksonville news anchor Courtney Cole tweeted photos of the scene.

    Continue reading
  • Everything you wanted to know about modern network congestion control but were perhaps too afraid to ask

    In which a little unfairness can be quite beneficial

    Systems Approach It’s hard not to be amazed by the amount of active research on congestion control over the past 30-plus years. From theory to practice, and with more than its fair share of flame wars, the question of how to manage congestion in the network is a technical challenge that resists an optimal solution while offering countless options for incremental improvement.

    This seems like a good time to take stock of where we are, and ask ourselves what might happen next.

    Congestion control is fundamentally an issue of resource allocation — trying to meet the competing demands that applications have for resources (in a network, these are primarily link bandwidth and router buffers), which ultimately reduces to deciding when to say no and to whom. The best framing of the problem I know traces back to a paper [PDF] by Frank Kelly in 1997, when he characterized congestion control as “a distributed algorithm to share network resources among competing sources, where the goal is to choose source rate so as to maximize aggregate source utility subject to capacity constraints.”

    Continue reading
  • How business makes streaming faster and cheaper with CDN and HESP support

    Ensure a high video streaming transmission rate

    Paid Post Here is everything about how the HESP integration helps CDN and the streaming platform by G-Core Labs ensure a high video streaming transmission rate for e-sports and gaming, efficient scalability for e-learning and telemedicine and high quality and minimum latencies for online streams, media and TV broadcasters.

    HESP (High Efficiency Stream Protocol) is a brand new adaptive video streaming protocol. It allows delivery of content with latencies of up to 2 seconds without compromising video quality and broadcasting stability. Unlike comparable solutions, this protocol requires less bandwidth for streaming, which allows businesses to save a lot of money on delivery of content to a large audience.

    Since HESP is based on HTTP, it is suitable for video transmission over CDNs. G-Core Labs was among the world’s first companies to have embedded this protocol in its CDN. With 120 points of presence across 5 continents and over 6,000 peer-to-peer partners, this allows a service provider to deliver videos to millions of viewers, to any devices, anywhere in the world without compromising even 8K video quality. And all this comes at a minimum streaming cost.

    Continue reading
  • Cisco deprecates Microsoft management integrations for UCS servers

    Working on Azure integration – but not there yet

    Cisco has deprecated support for some third-party management integrations for its UCS servers, and emerged unable to play nice with Microsoft's most recent offerings.

    Late last week the server contender slipped out an end-of-life notice [PDF] for integrations with Microsoft System Center's Configuration Manager, Operations Manager, and Virtual Machine Manager. Support for plugins to VMware vCenter Orchestrator and vRealize Orchestrator have also been taken out behind an empty rack with a shotgun.

    The Register inquired about the deprecations, and has good news and bad news.

    Continue reading
  • Protonmail celebrates Swiss court victory exempting it from telco data retention laws

    Doesn't stop local courts' surveillance orders, though

    Encrypted email provider Protonmail has hailed a recent Swiss legal ruling as a "victory for privacy," after winning a lawsuit that sees it exempted from data retention laws in the mountainous realm.

    Referring to a previous ruling that exempted instant messaging services from data capture and storage laws, the Protonmail team said this week: "Together, these two rulings are a victory for privacy in Switzerland as many Swiss companies are now exempted from handing over certain user information in response to Swiss legal orders."

    Switzerland's Federal Administrative Court ruled on October 22 that email providers in Switzerland are not considered telecommunications providers under Swiss law, thereby removing them from the scope of data retention requirements imposed on telcos.

    Continue reading
  • Japan picks AWS and Google for first gov cloud push

    Local players passed over for Digital Agency’s first project

    Japan's Digital Agency has picked Amazon Web Services and Google Cloud for its first big reform push.

    The Agency started operations in September 2021, years after efforts like the UK's Government Digital Service (GDS) or Australia's Digital Transformation Agency (DTA). The body was a signature reform initiated by Prime Minister Yoshihide Suga, who spent his year-long stint in the top job trying to curb Japan's reliance on paper documents, manual processes, and faxes. Japan's many government agencies also operated their websites independently of each other, most with their own design and interface.

    The new Agency therefore has a remit to "cut across all ministries" and "provide services that are driven not toward ministries, agency, laws, or systems, but toward users and to improve user-experience".

    Continue reading
  • Singaporean minister touts internet 'kill switch' that finds kids reading net nasties and cuts 'em off ASAP

    Fancies a real-time crowdsourced content rating scheme too

    A Minister in the Singapore government has suggested the creation of an internet kill switch that would prevent minors from reading questionable material online – perhaps using ratings of content created in real time by crowdsourced contributors.

    "The post-COVID world will bring new challenges globally, including to us in the security arena," said Minister for Defence Dr Ng Eng Hen at a Tuesday ceremony to award the city-state's 2021 Defense Technology Prize.

    "For operations, the SAF (Singapore Armed Force) has to expand its capabilities in the digital domain. Whether for administrative or operational purposes, I think that we will need to leverage technology to the maximum," he declared.

    Continue reading

Biting the hand that feeds IT © 1998–2021