Opinion What happens when your data sample isn’t representative of the public? We already know, for 2015 is turning out to be the year of the dodgy opinion poll. And it isn’t just politicians who are being stung.
In Windows 10, Microsoft says it will hide running apps from the task bar, breaking a twenty-year-old convention which started with Windows 95 – potentially causing confusion for hundreds of millions of regular users.
It’s doing so because fanbois who signed up for Microsoft's public Windows preview programme say they marginally preferred the change.
Let’s leave aside for whether the change will cause confusion or not, and examine how and why the decision was made. Last week, Microsoft’s Gabe Aul wrote that the Windows 10 designers had listened to the self-selecting “Insiders” in making the choice, and treated the feedback from the sample as if it was scientifically valid. One option had a satisfaction rating of 4.2, the other 3.8.
"We also observed that users are 34 per cent more likely to be strongly satisfied with the filtered Taskbar and three times less likely to be strongly dissatisfied compared to the global task bar."
And that’s the problem right there. The self-selecting sample might not actually be representative of the views of the regular Joe or Joleen.
This year, two national elections – the UK and Poland – have sprung major “surprises” that defied all the opinion pollsters. In both cases, it turned out the people the pollsters sampled weren’t giving them the full picture.
In the UK, it seems, people withheld their true intentions from the pollsters, although other factors were in play too. Every expert predicted a hung Parliament and yet the Conservatives won a small but definitive majority.
In Poland, only two out of 22 of the most recent polls gave the eventual winner (Duda) a lead. The defeated president had led the polls since September. The pollsters oversampled metropolitan areas, understating the eventual winner's support among rural voters, and turnout was high. His victory “surprised the party itself” – but then they’d been listening to pollsters more than their own supporters.
In both countries, the poll sample did not reflect the views of the population. Too much weight was given to data that didn’t support the conclusions that were then drawn from it. Microsoft has exposed itself to the same peril with its Insider programme.
Opening the Windows Insider programme to anyone who wanted to join had merit as a bug-squashing exercise. But it was also more than that: it demonstrated Microsoft was in touch with its user base.
After the years of the Sinofsky rampage, which via Windows 8 forced drastic changes on to users, Microsoft felt it needed to be more of a listening company. Windows Insider was a “listening exercise”. It showed Microsoft now cared much more about the user experience; design could be democratic.
The specific problem here is quite straightforward. Normal people don’t sign up in large numbers to try out very rough alpha software, or at least not knowingly. This means only the most devoted fanbois and developers have been using early builds of Windows 10, which has been very much an Alpha quality release until last week, when a much more mature build arrived.