Hoarding data in underground bunkers
Live engine data is sent back to the Lotus F1 headquarters at Enstone and then on to Mercedes in Northampton. There is around 400ms of latency in the link. While teams are guaranteed bandwidth over the standard MPLS connection at different circuits, the latency can vary from 25ms to 250ms.
Looking for the man with the chair and the white cat
This bunker is referred to as Teletubby Land: it is grassed over on top and often you can find rabbits on the roof. It was opened in 2008 to house Lotus F1's new super computer cluster. The reason it was built mainly underground was because of planning permission. Smith said: "We are in a beautiful part of the Cotswolds, the only way we could build it was to hide it as much as possible. Our actual site used to be a quarry, so nothing goes above tree height. In fact it is easy to drive straight past us, you wouldn't know we were here, apart from the sign on the gate."
All teams have to use the same McLaren Applied Technologies ECU and analysis software – called Atlas – but Lotus uses its own suites of in-house tools to manage the car set-up and for further analysis. Smith explained: "We have our own tools for data mining and are working with our technical partners, Microsoft and EMC to progress on this, from both a hardware and software side."
Its statistics software generates more than 2,500 statistics per car, per lap, which the engineers use to analyse the performance of the car and the driver. It also feeds the live telemetry data into more mathematical models to produce virtual channels, which can be viewed in real time as the car is running, effectively doubling the amount of information.
Development of the software is an on-going project with the teams growing and shrinking. Last year, 20 developers worked on the software.
The tools are mainly used by race and performance engineers. These are the people that decide on the setup of the car as well as giving the direction Lotus should be developing the car in. Short-term changes at the track are decided by the engineers there, on a race-to-race basis, but the team pores through years of data to see how it can improve for the next iteration of the car.
Back at base there are two huge data centres, again using VCE Vblock systems – in this case the larger Vblock 320. There is full redundancy, with the data centres acting as an active-active pair. Either one can run the whole site. The applications a Lotus employee is actually using can move from one side of the site to the other dynamically, without any intervention or the user noticing; handy for any system maintenance.
To keep up with the rapid growth of the data sets Lotus captures, data is archived in a pair of EMC Atmos archive systems, again with one in each data centre. These automatically migrate off data that has not been referenced, leaving a stub in the main system and a rapid index to the storage, where most of the petabytes of data are held.
Lotus does use the cloud, but Smith reckons he has the luxury of having sufficient compute power needed on premises, in his own private cloud. The initial move was office systems – Office 365 last year – with the company in the process of moving some of its other applications and big-data projects. The benefit? "Flexibility to call on compute power when needed," he said.
While the races produce a lot of data, it's the computational fluid dynamics (CFD) from the wind tunnel that consume the majority of the massive amounts of compute power Lotus has at its disposal, creating roughly 10TB of fresh data per week.
To stop teams from entering a spending war, the Formula 1 rules dictate not only the maximum teraflops a team can run for its CFD simulations, but also that wind tunnel models are a maximum of 60 per cent of full size.
Bigger wind tunnels get better results, but the cost to build and run them goes up enormously. As it is, the 60 per cent tunnel that Lotus runs is powered by a 3,000hp motor – nearly four times the power of the real car.
The attention to detail in the IT seems exceptional to an outsider, but it's not really out of place in the Formula 1 environment, where the engineering is honed to the finest possible levels. Simple elements of the car are 3D printed or made out of single billet blocks, by drilling, machining and spark erosion.
It's not just the parts of the car which are an engineering joy to handle; all the tools which are used to make the parts are similarly wonderful. You have to wonder about the tools used to make the tools.
Anthony Smith is Lotus F1's converged infrastructure specialist
So next time you watch a Grand Prix, look out for Lotus cars and think about what is going on in the background and the trackside IT support to keep things running and put Lotus' cars in the purple.®