This article is more than 1 year old

Is Britain really worse at 4G than Peru?

Crowd or non crowd - network industry rages on numbers

Is time a good performance proxy?

OpenSignal doesn’t hide the fact that it uses time as a proxy for performance. In its reports, it explains: "Rather than measure geographic coverage, OpenSignal's availability metric tracks the proportion of time users have access to a particular network. For example if a country has 50 per cent 4G availability, then on average that country's 4G users can find an LTE signal half of the time."

It also offered a detailed explanation of how it deals with biases. One of these is hotly contested, and critics say it may account for the “worse than Peru” claims in the papers this week.

In emerging economies, a metropolitan population is packed in urban areas, but the disparity between rural and urban is far greater than in developed markets. In rural areas, there may be 2G or no coverage at all.

“I guess it’s possible you find more technophiles in cities than rural areas but I’m not sure that’s really true. It’s hard to be absolutely sure. I can’t think of a good case even with lots of technophiles it would introduce a bias sufficiently serious to bring them into question,” says OpenSignal's Webb.

“We’ve looked at India and we don’t see any bias in that respect. We don’t get a skew,” he says.

“It would be too obtrusive to ask for every tester's income and their age. And that would probably breach data protection regulations. But every analysis we’ve done of rural vs urban doesn’t show any bias. It could change but we think it’s sufficiently unbiased so we’re generating answers that are pretty good.”

OpenSignal has examined education and income of its users on subsets of London data, and found no bias.

Back to the UK, comparing areas OpenSignal’s crowd and RootMetrics’ field data have both covered, such as London, finds better network performance numbers in the crowd study. RootMetrics’ Stonham thinks this is because of the area covered.

"They define London as more of an urban area. Our London 'Metro' area goes from Southend to Slough, from Tunbridge Wells to Cambridge, and it’s a large area with a strong mix of urban and rural areas. What find is in our London ‘Urban’ area, measured inside the North and South circulars, the LTE percentage is higher.

“We tried comparing major cities such as Madrid, New York and London, and it’s very difficult. Even with a rigorous scientific approach, there were limits we had to bear in mind.”

GWS’s criticism of Which? magazine’s “worse than Peru” phrase is its relatively small sample size.

“They had 30,000-ish app users and 500 million data points in the UK, but only 5,000 people in Peru with 5 million data points. You’re comparing drastically different numbers. That’s 1,200 data points per person,” a GWS spokesperson told us. Because it’s a background test and data allowances are relatively much lower in Peru, and other emerging markets, he doubts an average user would tolerate large chunks going on background tests.

Webb explains that OpenSignal is confident if just 100 app users are gathering data. The firm derives its confidence intervals using CLT (Central Limit Theory) “which argues that the mean of any collection of random variables (subject to minor technical constraints our data can readily be shown to satisfy) will tend toward a Gaussian distribution,” the firm explains. “Further it says that the variance of this mean will equal the variance of the underlying population divided by the sample size. Hence, if the CLT holds, we can derive the variance of our mean from the variance of the underlying data set. It is important then to show that the CLT holds for our data and for the sampling sizes we normally use.”

Webb is also aware of the biases often raised against crowdsourced data.

“We suspect operators may be getting their employees to do tests in good signal areas. So we end up with 1 average sample per user. We then average these across the operator base, then remove top five per cent and bottom five per cent of the user base.”

RootMetrics still uses some crowd data as an overlay on scouting data, and GWS has an app for anyone who to test their network speed. Both firms are sticking to the controlled approach. Stonham, who made the journey away from the evangelizing crowd, told us:

“Sometimes I feel crowd data is used quite emotionally. ‘It’s where people are so it must be true.’ But this is like believing what you hear on the news: it’s an echo chamber, it’s a self fulfilling prophecy.” ®

More about

TIP US OFF

Send us news


Other stories you might like