Google's plan to make User-Agent string even less useful breaks our device detection tech, says NetMarketShare

Outfit quits browser stats game, and you'll never guess who benefits most from shift to Client Hints

NetMarketShare – which has supplied free statistics on browsers, devices, operating system, and search engines for the last 14 years – is ending its reports, with October 2020 being the last month covered.

In a notice on its site, the California-based company said: "We are retiring NetMarketShare in its current form. October, 2020 is the last month of data. All billing for existing accounts has been stopped. All outstanding balances are being refunded."

The primary reason given is that "an upcoming change in browsers will break our device detection technology and will cause inaccuracies for a long period of time."

Secondly, it said that bot detection is "increasingly difficult" and that it did not want to accept "increasing levels of inaccuracy." It is promising to re-emerge at some unspecified future date.

The October 2020 report is to be NetMarketShare's last

The stats were based on data from users of its Net Applications analytics and social bookmarking products, which the company said gives it around 100 million valid sessions per month over thousands of websites. NetMarketShare explained that data gathered from web servers is more accurate than surveys (self-selected) or ISP data (regional bias), though the figures are still vulnerable to bias from the make up of its customer base.

That said, NetMarketShare has reported similar figures to its rival Statcounter. For example, for October 2020 NetMarketShare's browser stats had Chrome at 65.02 per cent, Safari at 18.25 per cent, Firefox at 3.39 per cent, and Edge (2.96 per cent) as its top four, where Statcounter had Chrome (66.12 per cent), Safari (17.24 per cent), Firefox (3.98 per cent), and Samsung (3.18 per cent), with Edge at 2.85 per cent. These are global market share figures on all device types.

Access to reliable statistics is valuable for both business and technical reasons, and is especially important to determine the extent of monopoly in browser, operating system, or device technology.

Guess who has the best stats?

There is no doubting who has the best statistics. Google benefits from the dominance of Chrome plus its 90-plus per cent search engine share, further bolstered by wide use of Google Analytics, which, according to W3Techs, is used "by 84.1 per cent of all the websites whose traffic analysis tool we know."

Aside from bots, the problem faced by NetMarketShare is the declining value of the User-Agent string included in the header data sent by a browser to a web server. It refers to this project from the Web Incubator Community Group (WICG), which includes the proposal that "Browsers should deprecate the User-Agent string over time, initially locking bits of its value, and ramping up over time to lock the entire string to something generic for the device type."

In its place, the group offers Client Hints (UA-CH), a set of name value pairs giving information including browser version, platform, CPU architecture, device model, viewport dimensions and pixel density, and device memory. Google is backing these proposals with information for developers. There are a few other notable features of Client Hints:

• Most Client Hints are not supplied by default, but only when the server requests them
• Most Client Hints are not sent on the first navigation request when a user visits a site
• Client Hints are only ever sent over secure connections (HTTPS)
• Client Hints are not available to JavaScript, though some may be surfaced by a userAgentData API. JavaScript developers are expected to "examine other parts of the exposed API surface" to determine compatibility, or to do so in server-side code

As the WICG notes: "Sites that wish to provide market share analytics using UA-CH will need to inspect the Sec-CH-UA header, that is sent by default on every request, and keep a record of it."

Developers can experiment with Client Hints here

Getting information from the browser has long been a problem since it is untrusted: a browser can send what it likes. Incorrect information is common, especially from browsers with small market share, as developers may otherwise declare a browser incompatible with their site or serve downgraded content. Another issue with providing detailed information is that it enables easier fingerprinting, where sites attempt to identify a user even when cookies are disabled or refused.

The current status of Client Hints is that it is in development in Chrome/Chromium. The latest status report said that "its timeline will be finalized once US-CH ships," though freezing the User-Agent string can take place before Client Hints are available.

In a discussion here, Google Developer Relations guy Yoav Weiss said the intent was to freeze the Chromium browser version in June 2020, but in fact Chrome 86 (the current version) still gives detailed version information.

Weiss said: "We expect this change to improve compatibility, as UA sniffing based on UA-CH is bound to be more reliable than the current status quo... we have other vendors on board with UA freezing, but not necessarily with the UA Client Hints mechanism, that is supposed to replace it.

"That can create a tricky situation, where developers would need to rely on the User-Agent string for some browsers and on UA-CH for others."

Hard for developers, and even harder for those trying to gather market share data. While there are good reasons for browsers to send less detailed information, making it harder for sites to collect browser statistics does happen to play into the hands of Google, which has many other ways to gather this information. ®