Waking up to a phone call in the wee small hours of the morning is almost never good. It’s usually a wrong number, a drunk ex wanting to talk, or the news that someone has died.
When NASA's Kepler space telescope mission manager Charlie Sobeck was woken by ringing at 0125 on April 8, it was close to the latter. The Kepler probe was one step from failure and NASA was rounding up its staff to perform a week-long session of multi-million-mile IT support.
The 2,320lb (1,052kg) telescope is orbiting the Sun 70 million miles from Earth, and for the past seven years has been scanning the skies for exoplanets orbiting other stars or wandering alone through the heavens.
The telescope was being reorientated for the K2 mission, which is looking for supernovas and star formations. In March, NASA commanded the telescope to shift its position for a series of shots using its 95 megapixel camera and prepared to receive data.
But on that fateful Friday last month, NASA found out that its distant instrument was stuck in emergency mode – the last stage before final shutdown. Its main computers were offline, no data was being recorded, and the spacecraft was burning fuel at a prodigious rate.
"As I recall, I received a call from the mission director at Ames, Marcie Smith, at 1:25 a.m. Friday morning. Knowing that there was a planned spacecraft contact, I expected that she would tell me that the spacecraft point was just a bit off, and we’d have to give it a nudge," Sobeck said.
"Instead I heard, 'We’re in Emergency Mode.' Within two minutes we confirmed what steps should be taken, and what resources needed to be immediately brought in, and that the flight team in Boulder had already begun the recovery actions. I headed into the office.”
Kepler is run by a joint team at NASA, manufacturers Ball Aerospace, and the Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado. By the time Sobeck got into the office that night everyone was either there or on the end of a phone line, and the team scarcely left the office for the next three days.
One step from disaster
The software running Kepler has a number of modes, including a safe mode if something goes wrong. But there’s also an emergency mode, never before activated on Kepler, that only kicks in if the ‘scope is in serious trouble.
Emergency mode is when the satellite thinks all of its key instruments are down. It shuts down its primary and secondary main computers, fires up two backup computers, and its thrusters put it into a slow spin designed to keep its solar panels orientated towards the Sun for maximum power.
You spin me right round baby, right round ... NASA's Kepler 'scope
That burns up the telescope’s limited fuel supplies and means Earth can only communicate with the onboard systems infrequently. All systems and instruments are switched off, apart from the thrusters, comms gear, and the primary and secondary backup computers, which are less able than the main systems.
The first stage was to reestablish contact with the telescope using the Deep Space Network of communications antennas in the US, Spain, and Australia. Time on the powerful radio network must be booked weeks in advance, but NASA declared a “space emergency” to get immediate access.
"We do not declare a spacecraft emergency when the spacecraft merely goes into Safe Mode, or if we simply don’t know what is going on," Sobeck said. "We use the spacecraft emergency card only when we truly believe the loss of the spacecraft is imminent without it."
Over the next three days, the Deep Space Network was focused on Kepler for about 20 hours a day. Kepler was rotating once every two hours, and its communications antenna could only maintain contact with the network back on Earth for twenty minutes per rotation.
In the time between contact, NASA needed the antenna array to communicate with its other satellites, giving the Kepler team a chance to get some sleep.