ICO orders release of (mostly useless) weather station data

CRU cuts of weather datasets released into the wild


The Information Commissioner's Office has ordered the University of East Anglia to release a portion of a weather dataset. The University's Climatic Research Unit had shared the data with Georgia Tech but refused to release it more widely. A leading Oxford physicist, Professor Jonathan Jones, made the successful request, which the ICO has now published.

The CRUTEM data set contains gridded weather station data provided by over 100 national meteorological services. Although CRU shared at least part of the data set with sympathetic academics, it had refused it to Jones and others in 2009, claiming several justifications. One justification that it was already available. Others were that the data wasn't already available but that making it available would breach copyright, harm the suppliers and jeopardise international relations.

The ICO rejected all grounds and ordered it to be disclosed to Professor Jones.

East Anglia had claimed that CRUTEM station records and the collection of records – the database – were both covered by copyright, the latter with a term of 15 years. UEA said it would harm its ability to commercially exploit the database. The ICO accepted that the copyright existed and pointed out that DEFRA guidelines should not prevent the data being disclosed. It added that CRU had been unable to demonstrate harm, or potential harm.

Does it matter? Not so much directly, one of the requesters explained last year. According to climate blogger Steve McIntyre, the real story was in the prolonged cover-up, and the improbable justifications, rather than anything nefarious in the code itself.

"... CRUTEM was an almost microscopically small issue in the Climategate emails – Climategate was about the Hockey Stick and its handling by IPCC, not CRUTEM. CRUTEM was mentioned in only 25 emails and, even then, often passim," explained McIntyre last year.

"My long-standing position on CRUTEM was that CRU's obstruction of data requests was most likely due to its desire to conceal that it did so little work on quality control; that the CRU result could be derived so trivially that, in effect, CRU no longer served any useful function in this field.

"The criticism from Climate Audit was that (1) CRU provided their station data as collated to 'friends' but not to potential critics; and (2) that their excuses for not providing station data were what one London reporter (not Jonathan Leake) described to me as 'deliberately deceptive'."

Dodgy science – but great literature

McIntyre's point about "negligiable quality control" was borne out in the famous HARRY_READ_ME.TXT file, a 90,000-word document recording attempts to update CRU's "flagship" product. Here a few choice excerpts:


"So.. we don't have the coefficients files (just .eps plots of something). But what are all those monthly files? DON'T KNOW, UNDOCUMENTED. Wherever I look, there are data files, no info about what they are other than their names. And that's useless..."

"Then got a mail from PJ [CRU director Phil Jones] to say we shouldn't be excluding stations inside 8km anyway – yet that's in IJC – Mitchell & Jones 2005! So there you go."


"[G]etting seriously fed up with the state of the Australian data. so many new stations have been introduced, so many false references ... so many changes that aren't documented. Every time a cloud forms I'm presented with a bewildering selection of similar-sounding sites, some with references, some with WMO codes, and some with both. And if I look up the station metadata with one of the local references, chances are the WMO code will be wrong (another station will have it) and the lat/lon will be wrong too."


"I am very sorry to report that the rest of the databases seem to be in nearly as poor a state as Australia was. There are hundreds if not thousands of pairs of dummy stations, one with no WMO and one with, usually overlapping and with the same station name and very similar coordinates. I know it could be old and new stations, but why such large overlaps if that's the case? Aarrggghhh! There truly is no end in sight."
"I am seriously worried that our flagship gridded data product is produced by Delaunay triangulation – apparently linear as well. As far as I can see, this renders the station counts totally meaningless. It also means that we cannot say exactly how the gridded data is arrived at from a statistical perspective – since we're using an off-the-shelf product that isn't documented sufficiently to say that ... Of course, it's too late for me to fix it too. Meh."

"OH FUCK THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm hitting yet another problem that's based on the hopeless state of our databases. There is no uniform data integrity, it's just a catalogue of issues that continues to grow as they're found."

In Parliament's enquiries into the Climategate Affair, Graham Stringer MP was surprised to learn that the CRU team couldn't produce the same result twice.

"When I asked Oxburgh if [Keith] Briffa [CRU academic] could reproduce his own results, he said in lots of cases he couldn't," Stringer told us. "That just isn't science. It's literature. If somebody can't reproduce their own results, and nobody else can, then what is that work doing in the scientific journals?" ®

Similar topics


Other stories you might like

  • Amazon warehouse staff granted second chance to vote for unionization

    US labor watchdog tosses previous failed result in the trash

    America's labor watchdog has given workers at Amazon’s warehouse in Bessemer, Alabama, another crack at voting for unionization after their first attempt failed earlier this year.

    “It is ordered that the election that commenced on February 8 is set aside, and a new election shall be conducted,” Lisa Henderson, regional director at the National Labor Relations Board, ruled [PDF] on Tuesday.

    “The National Labor Relations Board will conduct a second secret ballot election among the unit employees. Employees will vote whether they wish to be represented for purposes of collective bargaining by the Retail, Wholesale and Department Store Union.”

    Continue reading
  • It's the flu season – FluBot, that is: Surge of info-stealing Android malware detected

    And a bunch of bank-account-raiding trojans also identified

    FluBot, a family of Android malware, is circulating again via SMS messaging, according to authorities in Finland.

    The Nordic country's National Cyber Security Center (NCSC-FI) lately warned that scam messages written in Finnish are being sent in the hope that recipients will click the included link to a website that requests permission to install an application that's malicious.

    "The messages are written in Finnish," the NCSC-FI explained. "They are written without Scandinavian letters (å, ä and ö) and include, for example, the characters +, /, &, % and @ in illogical places in the text to make it more difficult for telecommunications operators to filter the messages. The theme of the text may be that the recipient has received a voicemail message or a message from their mobile operator."

    Continue reading
  • AsmREPL: Wing your way through x86-64 assembly language

    Assemblers unite

    Ruby developer and internet japester Aaron Patterson has published a REPL for 64-bit x86 assembly language, enabling interactive coding in the lowest-level language of all.

    REPL stands for "read-evaluate-print loop", and REPLs were first seen in Lisp development environments such as Lisp Machines. They allow incremental development: programmers can write code on the fly, entering expressions or blocks of code, having them evaluated – executed – immediately, and the results printed out. This was viable because of the way Lisp blurred the lines between interpreted and compiled languages; these days, they're a standard feature of most scripting languages.

    Patterson has previously offered ground-breaking developer productivity enhancements such as an analogue terminal bell and performance-enhancing firmware for the Stack Overflow keyboard. This only has Ctrl, C, and V keys for extra-easy copy-pasting, but Patterson's firmware removes the tedious need to hold control.

    Continue reading

Biting the hand that feeds IT © 1998–2021