Rewriting the checklists: 50 years since Apollo 13 reported it 'had a problem' – and boffins saved the day

It's instrumentation. It must be instrumentation. It wasn't instrumentation. Final part of The Register's look at Apollo 13


Part two 55 hours, 52 minutes and 58 seconds into Apollo 13's mission, capsule communicator Jack Lousma* asked the crew to stir the spacecraft's cryo tanks. Command Module Pilot Jack Swigert did so, and the "boring" mission became suddenly all too interesting.

Writing in his book Failure Is Not An Option Gene Kranz recalled Sy Liebergot, responsible for the electrical, environmental, communications, cryogenic and fuel cell systems of the spacecraft - aka EECOM - made the request.

A veteran controller, Liebergot was dealing with two issues aboard the spacecraft. An alarm had sounded in the capsule concerning hydrogen pressure in tank 1 and its warning light was illuminated both on his console at Mission Control (MCC) and in the spacecraft itself.

Apollo 13 was on the translunar cruise. The lunar module had been collected from the S-IVB and the team enjoyed continuous voice and data communications. Liebergot, however, had a second glitch to deal with – oxygen tank 2 had been reading normal at 80 per cent during the mission but the telemetry has recently been a bit odd.

"During our shift," recalled Kranz, "the gauge went through four rapid up-and-down cycles, finally failing and sticking at a constant reading of 100 per cent."

The team no longer had a valid reading from the sensor.

Cryogenic oxygen and hydrogen mixed and reacted in Apollo's three fuel cells to create electrical power, as well as provide water for the crew to drink and also cool the systems of the Command and Service Module (CSM). Loaded as a super-cold liquid, the cryos were by now part gas and part liquid.

The mixture could be stirred up by fans activated by the crew in order to get a precise measurement of quantity. There were also heaters in the tanks to raise the pressure.

Keen to avoid the crew being awoken by another alarm, Liebergot made the request for a stir, which the capcom passed on to Jack Swigert 55 hours, 52 minutes and 58 seconds into the mission.

NASA's chronology of events for the seconds between Swigert hitting the button and his infamous radio call to MCC of "Okay, Houston, we've had a problem here" is harrowing. Amid electrical "disturbances" the pressure in Oxygen tank 2 began to rise, hitting a maximum of 1,008.3 psia.

The quantity jumped to off-scale high before falling due to a suspected failed sensor. The temperature of tank 2 went off-scale low as another sensor likely failed.

At 55 hours, 54 minutes and 53 seconds, NASA noted "X, Y, and Z accelerations in CM indicate 1.17g, 0.65g, and 0.65g."

A tenth of a second later, the agency reckoned that tank 2 lost pressure and, in a marvelous bit of understatement, "the panel separated." In his book Flight director of Flight Operations Chris Kraft put it more bluntly: "One of the six compartments in the service module had been blown away."

Not, of course, that anyone in space or on the ground was aware of it at the time.

The Crisis Begins

Famously, ground controllers were initially disbelieving. As the Apollo 13 crew reported a litany of warning indications, Kranz's first thought was "The reports and our experience indicated an electrical glitch. I believed that we would quickly nail the problem and get back on track."

"I was wrong."

Kranz recalled it took a full 15 minutes before controllers understood the scope of the crisis that was unfolding, and that - far from putting boots on the Moon for a third time - the mission had changed into one of survival.

"It took extra seconds sorting out what was real and credible," recalled Kranz. The spacecraft appeared to be rapidly losing oxygen, which would take out the fuel cells. The loss of the fuel cells meant no electrical power and no control over the main propulsion system. "Nothing remotely like this had ever happened in simulation."

"Five minutes after the event, the significance of the crew's words, 'we had a pretty large bang...' hit me", Kranz remembered. He had been the flight director for Apollo 9 and the pyrotechnic shock when the CSM and S-IVB booster separated closed the fuel valves. The bang experienced by the Apollo 13 crew must have been awfully "solid," as Kranz put it.

As crew and controllers became increasingly frustrated with the impossible failures that should have been precluded by the design of the Apollo spacecraft, the MCC team continued to do its stuff. Gary Scott, NASA's chap responsible for communications, had the spacecraft fall back to the less powerful Omni antennas to keep the link between the Earth and Apollo 13 open as the spacecraft was pushed by a (then) unknown force. Others continued wrestling with the problems.

Most the load seemed to rest of Liebergot's shoulders as he worked to staunch the hemorrhaging of fuel cell reactants.

It took Lovell, glancing out of a Command Module (CM) window, and informing MCC that the spacecraft was venting a gas into a space that "all the pieces of the puzzle came together," according to Kranz.

"I was damn angry that I had wasted 15 precious minutes by not assembling the pieces earlier," Kranz reproached himself. There would be no Moon landing this time, the sole objective was one of survival.

Cruising Home

Just over an hour after the explosion, and with 15 minutes of CM power remaining, the MCC team instructed the crew to make their way into the Lunar Module (LM). Despite being only designed for a 45-hour lifetime, the LM would have to take on lifeboat duties for considerably longer. The CM, required for re-entry into the Earth's atmosphere, would need to be powered down to conserve what was left of its batteries.

Chris Kraft arrived in Mission Control, and Kranz told him: "Chris, we're in deep shit." Liebergot confirmed that fuel cells 1 and 3 were dead, and the trajectory team reported on the abort options. Apollo 13 was 200,000 miles from Earth and 45,000 from the surface of the Moon.

There were two basic options. The fastest direct abort would require a jettison of the Lunar Module and firing the main engine; using all the fuel to get back in less than two days.

The other option, not using the main engine, would see Apollo 13 go around the Moon. The lunar module could then be kept, but even using the standard powered-down checklists, the battery of the two-crew spacecraft would be dead well before the crew made it home.

Kranz vividly describes the disagreements over the decision, with several favouring the option to get the crew home fastest. However, the argument that the CSM's propulsion system could not be trusted won out, and the team set about eking as much resource as possible out of the remaining systems. Anything not deemed essential (even the computer) was turned off.

The lunar module, as it turned out, made for a far better lifeboat than anyone could have reasonably expected. Docked to the dead weight of the CSM, it took the three crew around the Moon, and its engine - designed only for the Lunar descent - was repurposed for trajectory adjustments. Five hours after the explosion, the crew fired the descent engine for 35 seconds to put the spacecraft back on a free return trajectory around the Moon.

Two hours after Apollo 13 passed behind the Moon, the descent engine was fired again for just under five minutes, cutting the hours needed to return home and fixing the landing time for 142 hours, while moving the landing point from the Indian Ocean to the Pacific, near Samoa.

Other challenges, such as executing a Passive Thermal Control (PTC) manoeuvre, described by Kranz as "like trying to thread a needle with bad eyesight," when performed with the LM's thrusters, and building a contraption to adapt carbon dioxide scrubbers from the CSM for use in the LM (famously, the two were not interchangeable) needed to be accomplished by the tired crew as they barrelled back toward Earth in the frigid lander.

While other teams worked on, and succeeded in solving, the challenge of keeping the crew alive on the journey back to Earth, another worked on how the CSM would be reactivated. Its batteries, partially depleted in the minutes after the explosion, would need to be charged as far as possible from the LM. Its thrusters, left unused while the LM took care of attitude and trajectory, would be required once the lander was jettisoned.

Back To Earth

Cold, tired and in need of reassurance, the crew received the procedures to reactivate Odyssey just over six hours before the process was due to begin. The team had managed to secure a surplus of power by throwing out the original checklists and reducing the equipment running on the lander to a bare minimum.

As it transpired, the stringent power-saving would leave 20 per cent of LM electrical power left and water saving saw the mission end with 9 per cent of electrical energy remaining.

"Generating options," said a proud Kranz, "was our business, and options remained as long as there was power, water, oxygen and propellant. My controllers kept finding options."

Coming back with the lunar module attached brought with it its own challenges, not least of which was a container of radioactive fuel. The canister was designed to remain intact during re-entry; the only question was one of where would it land?

Initial calculations had it hitting a populated area of Madagascar. Then it looked like it was headed for deep water off New Zealand. As it transpired, a last-minute error in attitude control sent it elsewhere; to the bottom of the Indian Ocean, much to the relief of the Atomic Energy Commission.

The crew of Apollo 13 were allowed to start powering up the Odyssey two hours early in order to get some heat into the spacecraft. However, as Kranz noted "the most chilling discussion came a few hours before entry, as the crew jettisoned the service module and then maneuvered to observe the damage."

Lovell and Haise noted that "there is one whole side of the spacecraft missing." The damage, from top to bottom, validated the decision not to try to use the CSM's main propulsion system, but Kranz had avoided discussion of damage to the command module itself and, crucially, the heat shield at its base.

"At a certain point," he said, "the human factor has accomplished all it can."

Lander and Service Module jettisoned; the crew was on their own for re-entry. A final warning that one of the three command module batteries would fail as the parachutes deployed was issued and the communications blackout began.

"I felt a sense of loneliness in the room," recalled Kranz. The crew were now completely on their own as the controllers watched the clock tick down to the moment when a signal from Odyssey would arrive and confirm its survival.

The timer reached zero. And went past zero with no communication. The controllers continued their attempts to contact the capsule.

Kranz sums up the atmosphere: "We were now almost a minute past the expected signal acquisition time. Still no response. Seconds turned in to minutes and minutes into infinity. A sinking feeling, almost a dread, filled the room. When the wall clock rolled past one minute, we wondered what the hell had gone wrong."

It wasn't until one minute 28 seconds after the expected time of acquisition that a downrange aircraft broke the tension: "ARIA 4 has acquisition."

The control room held back from the partying. Next the parachutes had to deploy, and the crew lifted safely to the aircraft carrier. As those parachutes blossomed, Kranz recalled "I find myself crying unabashedly, then I try to suck it in, realising this is inappropriate, but it doesn't work; it only gets worse."

As the crew were handed over to the care of the commander of the aircraft carrier taskforce, for Mission Control it was over. Kranz: "We – crew, contractors, controllers – had done the impossible. The human factor had carried the day."

The crew were recovered by the USS Iwo Jima and the landing time was 17 April 1970 at 18:07:41 UTC.

Further reading

We could not cover every player and every twist of the Apollo 13 story and would suggest those interested seek out the wealth of writing on the mission.

We recommended the books of Lovell, Kluger, Cooper, Perrow, Kranz and Kraft in part one. Sy Liebergot's book Apollo EECOM also provided very useful in this part.

We must also repeat the recommendation to check out season 2 of Kevin Fong's 13 Minutes To The Moon podcast. And that Tom Hanks film? We'd recommend watching it with Jim Lovell's commentary running. ®

* NASA's chronology erroneously listed the request as issued by Apollo 11 CAPCOM Charlie Duke.

Similar topics


Other stories you might like

  • IPSE: More than a third of freelancers have quit contracting since IR35 reforms

    Exodus, movement of the people... to the Middle East or elsewhere

    More than a third (35 per cent) of contractors in the UK have become permanent employees, retired, shifted to work overseas or are "simply not working" since IR35 tax legislation was revised earlier this year.

    This is according to the Association of Independent Professionals (IPSE) which found 35 per cent fewer freelancers among those it surveyed since 6 April when the government pushed through the delayed reform.

    "This research shows the devastating impact the changes to IR35 have had on contractors, needlessly compounding the financial damage of the pandemic," said Andy Chamberlain, director of policy at IPSE. "Now, just when contractors are needed the most - amid mounting labour shortages across the UK and particularly in haulage - government decisions have drive out a third of the sector."

    Continue reading
  • New Relic guzzles down CodeStream to help devs jump straight from app error telemetry to offending code

    'I can debug production from the IDE,' said CS boss Peter Pezaris

    Observability company New Relic has acquired CodeStream, specialists in developer collaboration, with the aim being to connect observability data with code in the development environment.

    CodeStream, founded in 2017 by Peter Pezaris, adds instant developer communication to coding environments. For example, a developer puzzling over some code written by a colleague can click next to that code, type a message to the other dev, and they will receive it either in the IDE if they happen to be working on the same project, or in a messaging tool such as Slack, complete with a reference to the code in question. They reply, and a discussion begins.

    Although it may seem a small thing, given that they could just use Slack (or any number of other messaging services) directly, the context and convenience makes it a worthwhile collaboration tool. CodeStream also integrates with pull requests from GitHub, GitLab, BitBucket, and issue management from Jira, Trello and others.

    Continue reading
  • Analogue tones of a ZX Spectrum Load set to ride again via podcast project

    Remember the R Tape Loading Error?

    The glory days of audio-cassette loading are set to return in the coming weeks, with retro fans to be treated to a broadcast for them to hit Play and Record to.

    Audio cassettes were the medium of choice for software back when Sinclair and Commodore's 8-bit hardware ruled the roost. The floppy disk seemed impossibly glamorous for the average home computer user and code was instead delivered via audio.

    While the sound of those files was unintelligible for most, for some enthusiasts it was possible to discern the type of data being loaded. Right up until the all-too-common R Tape Loading Error (which usually seemed to come right at the end of a lengthy period staring at a loading screen).

    Continue reading

Biting the hand that feeds IT © 1998–2021