Flexible friend: Data's Big digital journey online

The analogue shift disguised its original purpose

Falling costs, cheap comms, mass media

It was also a feature of most of these early media and channels that the apparatus needed was beyond the means of the individual and had to be centralised. Centralisation is a word that inevitably raises all kinds of ethical questions. When the means to copy or disseminate information are in the hands of those who have power, be it political, financial or religious, then objectivity goes out of the window.

Of course the situation was disrupted when the falling cost of technology made one medium or channel after another affordable to the individual. The disruption broke monopolies and allowed alternative points of view to be expressed. Things that were being concealed were revealed and official explanations could be queried. Possibly the greatest legacy of the IT explosion has been the disruption of questionable organisations and activities and it has been very rewarding to have been a part of it.

Possibly the greatest legacy of the IT explosion has been the disruption of questionable organisations and activities

In what was at the time seen as a quite distinct art, mathematicians and logicians were formulating the language of logic and sets. Boolean algebra, Venn diagrams, Carroll diagrams and Galois fields all helped to formalise thinking, at first human thinking, of course. It was not until the 1930s that Claude Shannon made the connection between the truth and falsehood of Boolean algebra and the binary signals of logic circuitry. Musical instruments have a long history and some developed to surprising levels of sophistication.

There were two basic problems as they grew in complexity. One of these was the need for a single player to control more sound-making devices and the other was to make the music more accessible by having automated players. In both cases the easiest instrument to work with was the pipe organ because the control system only needed enough power to operate air valves.

As organs were sounded by air blowers, which before electricity were hand-pumped, the mechanism of the self-playing organ would also be air-operated and the music would be stored on punched cards where the holes would control the admission of air.

Punched cards would also be adopted to control knitting machines and other early CAM devices as well as evolving into the punched tape and Hollerith cards used in tabulators and then computers. The invention of PCM by Alec Reeves allowed linear signals like audio and video to be expressed as binary data.

The second half of the twentieth century saw a joining up of all the dots, which became fantastically rapid towards the end. Automated servo-mechanisms developed for other purposes such as aerospace were applied to data storage devices so that the access to data needed no human intervention. Location of the required data was by supplying an address which operated the mechanism.

The rotating magnetic drum and then the rotating magnetic disk became the mainstay of these random-access data storage systems which allowed large amounts of data to be online for the first time. The combination of data transmission and computing allowed networks to be built, so that it was no longer necessary to be where the data were stored in order to access them.

Error correcting systems based on the mathematics of logic and Galois fields ensured that the data recovered from storage media and delivered over networks was error free. For the first time the information quality had nothing to do with the characteristics of the storage or transmission medium. In the absence of generation loss, where a copy is a clone, one of the distinctions between consumer and professional equipment evaporated and that would be disruptive.

Digital audio may not have been the first disruptive technology, but it does illustrate very nicely many of the principles. The sound quality of the Compact Disc buried everything that had gone before and it was an enormous and lasting success, continuing to sell to this day, 33 years after its introduction. Record companies had embraced a wear-free format that stored data using error correction and therefore suffered no generation loss. When the DAT format came along, which would allow the consumer to clone CDs, they lost control of their bowels.

Really, record companies? You're going to get the DAT format banned?

One of the ways to identify a disruptive technology is to watch for the knee-jerks of the threatened dinosaur. Frantic legal action resulted in the DAT format being banned. Hundreds of man-years of R & D went down the toilet and the revenge was polite, elegant and perfect. The recordable CD was simply brought forward. There wasn’t a darned thing the record companies could do about it as it adhered to a standard format. Touché.

A digital storage or transmission device doesn’t need to be source-specific as all binary data look the same. Clearly in a world of source-specific media, that would also be disruptive. Record companies failed to see that PCM audio data could also be delivered down data networks, but then the horizon does become very close when you have your snout in a trough. Having created tremendous accrued resentment in consumer and artiste alike, no life belts were thrown when they slipped beneath the waves.

Before the end of the last century, most information was recorded on source-specific linear media that needed human intervention and could not be put on line. Around the turn of the century this changed and now more data are on-line than not. Some people define big data as just lots of bytes and the attendant problems of handling them, but others think it is the opportunity for data mining, which of course is only possible when it’s all on line and networked. Both may be right.

Next time, I’ll look at early steps in information technology. ®

Big Data and All That is John Watkinson's look at the technical history of computing and the impact on society of the IT revolution.