White Screen of Death: Admins up in arms after experimental Google emission borks Chrome

Change rolled back, but it's not a good look

An experimental feature silently rolled out to the stable Chrome release on Tuesday caused chaos for IT admins this week after users complained of facing white, featureless tabs on Google's massively popular browser.

The issue affected thousands of businesses' terminal servers, with multiple users on the same server experiencing "white screen of death" at the same time.

Someone posting on the Chromium bug tracker mailing list described the problem as follows:

We have confirmed and replicated; when any user on a shared session citrix box locks their screen, all Chrome windows stop rendering ("White screen of death") until ANYONE unlocks their screen, upon which, all Chrome windows resume rendering. This looks like random behaviour to the user but we have confirmed lock/unlock is the culprit.

The person added: "We have fixed this temporarily by starting chrome with –disable-backgrounding-occluded-windows," applying the fix through a group policy object.

Google software engineer David Bienvenu jumped in to explain:

The experiment/flag has been on in beta for ~5 months. It was turned on for stable (e.g., m77, m78) via an experiment that was pushed to released Chrome Tuesday morning.

At 1824 UTC last night, Bienvenu rolled back the experiment change, noting "I'm not sure how long it takes to go live, but once it's live, users will need to restart Chrome to get the change."

This prompted one admin to snap back:

I am stunned by your response ... Do you see the impact you created for thousands of us without any warning or explanation? We are not your test subjects. We are running professional services for multi million dollar programs. Do you understand how many hours of resources were wasted by your "experiment"? Not acceptable..

Irate IT admins posting on the Chromium dev blog pointed out the problem didn't only affect Citrix, though with its wide adoption it was the highest-profile casualty. One posted:

We did not update to the latest version 78, we have had version 77.0.3865 in our Citrix PVS image since 10/24 and within the last couple of days we started getting the white screen issue. So the browser without being updated to a newer version must have gotten some kind of update "experiment" without updating the version. This also has affected our call center agents that use a WebRTC VOIP phone, and caused many IT folks to bang their heads for over a day now. I would be very interested to know when this is rolled back and how to turn these updates off so that I can roll out a new version in our image, test it in preprod and KNOW that it is not going to change until I change it.

A US school IT manager threatened to ditch Chrome OS in favour of Microsoft's Office 365 as a direct result of the silent experiment:

We're a large school district with 2,000 managed Enterprise Browsers and over 11,000 Chrome OS devices. We specifically do version control via the Admin Console and a central update server for applications to explicitly avoid these scenarios. If you're going to make configuration changes in the background rather than sticking to software revisions to make changes, perhaps we should be turning our eye toward Office 365 and another competitor browser for our environment as our helpdesk ticketing service and support lines have been inundated with this issue since the change went live Tuesday night and the only thing that visibly changed for us was the monthly updates from Microsoft... so we thought.

A poster on the Chromium bug report thread said they had installed Firefox as a workaround for silent Chrome updates that didn't increment the version number or give any other indication that today's build wasn't the same as yesterday's.

For those who are no longer keen to be part of the experiments, Peter Beverloo, who works on Chromium Web Capabilities, has listed that the Chromium Command Line --no-experiments parameter should work to opt out.

As for the "occluded windows" experiment in question, the change has been rolled back. Nonetheless, the customer confidence impact of discovering that what should be a stable build with updates controlled by local admins is no such thing will reverberate. No sysadmin wants to find that vendors are silently messing with what used to be a stable deployment configuration.

As one IT pro put it not one hour ago: "I don't think we should stop making noise about this. The issue now is that Google has gotten so big that they aren't concerned at all about what they have done because they know we will keep using their software." ®