Rewriting the checklists: 50 years since Apollo 13 reported it 'had a problem' – and boffins saved the day

It's instrumentation. It must be instrumentation. It wasn't instrumentation. Final part of The Register's look at Apollo 13

Part two 55 hours, 52 minutes and 58 seconds into Apollo 13's mission, capsule communicator Jack Lousma* asked the crew to stir the spacecraft's cryo tanks. Command Module Pilot Jack Swigert did so, and the "boring" mission became suddenly all too interesting.

Writing in his book Failure Is Not An Option Gene Kranz recalled Sy Liebergot, responsible for the electrical, environmental, communications, cryogenic and fuel cell systems of the spacecraft - aka EECOM - made the request.

A veteran controller, Liebergot was dealing with two issues aboard the spacecraft. An alarm had sounded in the capsule concerning hydrogen pressure in tank 1 and its warning light was illuminated both on his console at Mission Control (MCC) and in the spacecraft itself.

Apollo 13 was on the translunar cruise. The lunar module had been collected from the S-IVB and the team enjoyed continuous voice and data communications. Liebergot, however, had a second glitch to deal with – oxygen tank 2 had been reading normal at 80 per cent during the mission but the telemetry has recently been a bit odd.

"During our shift," recalled Kranz, "the gauge went through four rapid up-and-down cycles, finally failing and sticking at a constant reading of 100 per cent."

The team no longer had a valid reading from the sensor.

Cryogenic oxygen and hydrogen mixed and reacted in Apollo's three fuel cells to create electrical power, as well as provide water for the crew to drink and also cool the systems of the Command and Service Module (CSM). Loaded as a super-cold liquid, the cryos were by now part gas and part liquid.

The mixture could be stirred up by fans activated by the crew in order to get a precise measurement of quantity. There were also heaters in the tanks to raise the pressure.

Keen to avoid the crew being awoken by another alarm, Liebergot made the request for a stir, which the capcom passed on to Jack Swigert 55 hours, 52 minutes and 58 seconds into the mission.

NASA's chronology of events for the seconds between Swigert hitting the button and his infamous radio call to MCC of "Okay, Houston, we've had a problem here" is harrowing. Amid electrical "disturbances" the pressure in Oxygen tank 2 began to rise, hitting a maximum of 1,008.3 psia.

The quantity jumped to off-scale high before falling due to a suspected failed sensor. The temperature of tank 2 went off-scale low as another sensor likely failed.

At 55 hours, 54 minutes and 53 seconds, NASA noted "X, Y, and Z accelerations in CM indicate 1.17g, 0.65g, and 0.65g."

A tenth of a second later, the agency reckoned that tank 2 lost pressure and, in a marvelous bit of understatement, "the panel separated." In his book Flight director of Flight Operations Chris Kraft put it more bluntly: "One of the six compartments in the service module had been blown away."

Not, of course, that anyone in space or on the ground was aware of it at the time.

The Crisis Begins

Famously, ground controllers were initially disbelieving. As the Apollo 13 crew reported a litany of warning indications, Kranz's first thought was "The reports and our experience indicated an electrical glitch. I believed that we would quickly nail the problem and get back on track."

"I was wrong."

Kranz recalled it took a full 15 minutes before controllers understood the scope of the crisis that was unfolding, and that - far from putting boots on the Moon for a third time - the mission had changed into one of survival.

"It took extra seconds sorting out what was real and credible," recalled Kranz. The spacecraft appeared to be rapidly losing oxygen, which would take out the fuel cells. The loss of the fuel cells meant no electrical power and no control over the main propulsion system. "Nothing remotely like this had ever happened in simulation."

"Five minutes after the event, the significance of the crew's words, 'we had a pretty large bang...' hit me", Kranz remembered. He had been the flight director for Apollo 9 and the pyrotechnic shock when the CSM and S-IVB booster separated closed the fuel valves. The bang experienced by the Apollo 13 crew must have been awfully "solid," as Kranz put it.

As crew and controllers became increasingly frustrated with the impossible failures that should have been precluded by the design of the Apollo spacecraft, the MCC team continued to do its stuff. Gary Scott, NASA's chap responsible for communications, had the spacecraft fall back to the less powerful Omni antennas to keep the link between the Earth and Apollo 13 open as the spacecraft was pushed by a (then) unknown force. Others continued wrestling with the problems.

Most the load seemed to rest of Liebergot's shoulders as he worked to staunch the hemorrhaging of fuel cell reactants.

It took Lovell, glancing out of a Command Module (CM) window, and informing MCC that the spacecraft was venting a gas into a space that "all the pieces of the puzzle came together," according to Kranz.

"I was damn angry that I had wasted 15 precious minutes by not assembling the pieces earlier," Kranz reproached himself. There would be no Moon landing this time, the sole objective was one of survival.

Cruising Home

Just over an hour after the explosion, and with 15 minutes of CM power remaining, the MCC team instructed the crew to make their way into the Lunar Module (LM). Despite being only designed for a 45-hour lifetime, the LM would have to take on lifeboat duties for considerably longer. The CM, required for re-entry into the Earth's atmosphere, would need to be powered down to conserve what was left of its batteries.

Chris Kraft arrived in Mission Control, and Kranz told him: "Chris, we're in deep shit." Liebergot confirmed that fuel cells 1 and 3 were dead, and the trajectory team reported on the abort options. Apollo 13 was 200,000 miles from Earth and 45,000 from the surface of the Moon.

There were two basic options. The fastest direct abort would require a jettison of the Lunar Module and firing the main engine; using all the fuel to get back in less than two days.

The other option, not using the main engine, would see Apollo 13 go around the Moon. The lunar module could then be kept, but even using the standard powered-down checklists, the battery of the two-crew spacecraft would be dead well before the crew made it home.

Kranz vividly describes the disagreements over the decision, with several favouring the option to get the crew home fastest. However, the argument that the CSM's propulsion system could not be trusted won out, and the team set about eking as much resource as possible out of the remaining systems. Anything not deemed essential (even the computer) was turned off.

The lunar module, as it turned out, made for a far better lifeboat than anyone could have reasonably expected. Docked to the dead weight of the CSM, it took the three crew around the Moon, and its engine - designed only for the Lunar descent - was repurposed for trajectory adjustments. Five hours after the explosion, the crew fired the descent engine for 35 seconds to put the spacecraft back on a free return trajectory around the Moon.

Two hours after Apollo 13 passed behind the Moon, the descent engine was fired again for just under five minutes, cutting the hours needed to return home and fixing the landing time for 142 hours, while moving the landing point from the Indian Ocean to the Pacific, near Samoa.

Other challenges, such as executing a Passive Thermal Control (PTC) manoeuvre, described by Kranz as "like trying to thread a needle with bad eyesight," when performed with the LM's thrusters, and building a contraption to adapt carbon dioxide scrubbers from the CSM for use in the LM (famously, the two were not interchangeable) needed to be accomplished by the tired crew as they barrelled back toward Earth in the frigid lander.

While other teams worked on, and succeeded in solving, the challenge of keeping the crew alive on the journey back to Earth, another worked on how the CSM would be reactivated. Its batteries, partially depleted in the minutes after the explosion, would need to be charged as far as possible from the LM. Its thrusters, left unused while the LM took care of attitude and trajectory, would be required once the lander was jettisoned.

Back To Earth

Cold, tired and in need of reassurance, the crew received the procedures to reactivate Odyssey just over six hours before the process was due to begin. The team had managed to secure a surplus of power by throwing out the original checklists and reducing the equipment running on the lander to a bare minimum.

As it transpired, the stringent power-saving would leave 20 per cent of LM electrical power left and water saving saw the mission end with 9 per cent of electrical energy remaining.

"Generating options," said a proud Kranz, "was our business, and options remained as long as there was power, water, oxygen and propellant. My controllers kept finding options."

Coming back with the lunar module attached brought with it its own challenges, not least of which was a container of radioactive fuel. The canister was designed to remain intact during re-entry; the only question was one of where would it land?

Initial calculations had it hitting a populated area of Madagascar. Then it looked like it was headed for deep water off New Zealand. As it transpired, a last-minute error in attitude control sent it elsewhere; to the bottom of the Indian Ocean, much to the relief of the Atomic Energy Commission.

The crew of Apollo 13 were allowed to start powering up the Odyssey two hours early in order to get some heat into the spacecraft. However, as Kranz noted "the most chilling discussion came a few hours before entry, as the crew jettisoned the service module and then maneuvered to observe the damage."

Lovell and Haise noted that "there is one whole side of the spacecraft missing." The damage, from top to bottom, validated the decision not to try to use the CSM's main propulsion system, but Kranz had avoided discussion of damage to the command module itself and, crucially, the heat shield at its base.

"At a certain point," he said, "the human factor has accomplished all it can."

Lander and Service Module jettisoned; the crew was on their own for re-entry. A final warning that one of the three command module batteries would fail as the parachutes deployed was issued and the communications blackout began.

"I felt a sense of loneliness in the room," recalled Kranz. The crew were now completely on their own as the controllers watched the clock tick down to the moment when a signal from Odyssey would arrive and confirm its survival.

The timer reached zero. And went past zero with no communication. The controllers continued their attempts to contact the capsule.

Kranz sums up the atmosphere: "We were now almost a minute past the expected signal acquisition time. Still no response. Seconds turned in to minutes and minutes into infinity. A sinking feeling, almost a dread, filled the room. When the wall clock rolled past one minute, we wondered what the hell had gone wrong."

It wasn't until one minute 28 seconds after the expected time of acquisition that a downrange aircraft broke the tension: "ARIA 4 has acquisition."

The control room held back from the partying. Next the parachutes had to deploy, and the crew lifted safely to the aircraft carrier. As those parachutes blossomed, Kranz recalled "I find myself crying unabashedly, then I try to suck it in, realising this is inappropriate, but it doesn't work; it only gets worse."

As the crew were handed over to the care of the commander of the aircraft carrier taskforce, for Mission Control it was over. Kranz: "We – crew, contractors, controllers – had done the impossible. The human factor had carried the day."

The crew were recovered by the USS Iwo Jima and the landing time was 17 April 1970 at 18:07:41 UTC.

Further reading

We could not cover every player and every twist of the Apollo 13 story and would suggest those interested seek out the wealth of writing on the mission.

We recommended the books of Lovell, Kluger, Cooper, Perrow, Kranz and Kraft in part one. Sy Liebergot's book Apollo EECOM also provided very useful in this part.

We must also repeat the recommendation to check out season 2 of Kevin Fong's 13 Minutes To The Moon podcast. And that Tom Hanks film? We'd recommend watching it with Jim Lovell's commentary running. ®

* NASA's chronology erroneously listed the request as issued by Apollo 11 CAPCOM Charlie Duke.

Similar topics

Other stories you might like

  • New York City rips out last city-owned public payphones
    Y'know, those large cellphones fixed in place that you share with everyone and have to put coins in. Y'know, those metal disks representing...

    New York City this week ripped out its last municipally-owned payphones from Times Square to make room for Wi-Fi kiosks from city infrastructure project LinkNYC.

    "NYC's last free-standing payphones were removed today; they'll be replaced with a Link, boosting accessibility and connectivity across the city," LinkNYC said via Twitter.

    Manhattan Borough President Mark Levine said, "Truly the end of an era but also, hopefully, the start of a new one with more equity in technology access!"

    Continue reading
  • Cheers ransomware hits VMware ESXi systems
    Now we can say extortionware has jumped the shark

    Another ransomware strain is targeting VMware ESXi servers, which have been the focus of extortionists and other miscreants in recent months.

    ESXi, a bare-metal hypervisor used by a broad range of organizations throughout the world, has become the target of such ransomware families as LockBit, Hive, and RansomEXX. The ubiquitous use of the technology, and the size of some companies that use it has made it an efficient way for crooks to infect large numbers of virtualized systems and connected devices and equipment, according to researchers with Trend Micro.

    "ESXi is widely used in enterprise settings for server virtualization," Trend Micro noted in a write-up this week. "It is therefore a popular target for ransomware attacks … Compromising ESXi servers has been a scheme used by some notorious cybercriminal groups because it is a means to swiftly spread the ransomware to many devices."

    Continue reading
  • Twitter founder Dorsey beats hasty retweet from the board
    We'll see you around the Block

    Twitter has officially entered the post-Dorsey age: its founder and two-time CEO's board term expired Wednesday, marking the first time the social media company hasn't had him around in some capacity.

    Jack Dorsey announced his resignation as Twitter chief exec in November 2021, and passed the baton to Parag Agrawal while remaining on the board. Now that board term has ended, and Dorsey has stepped down as expected. Agrawal has taken Dorsey's board seat; Salesforce co-CEO Bret Taylor has assumed the role of Twitter's board chair. 

    In his resignation announcement, Dorsey – who co-founded and is CEO of Block (formerly Square) – said having founders leading the companies they created can be severely limiting for an organization and can serve as a single point of failure. "I believe it's critical a company can stand on its own, free of its founder's influence or direction," Dorsey said. He didn't respond to a request for further comment today. 

    Continue reading
  • Snowflake stock drops as some top customers cut usage
    You might say its valuation is melting away

    IPO darling Snowflake's share price took a beating in an already bearish market for tech stocks after filing weaker than expected financial guidance amid a slowdown in orders from some of its largest customers.

    For its first quarter of fiscal 2023, ended April 30, Snowflake's revenue grew 85 percent year-on-year to $422.4 million. The company made an operating loss of $188.8 million, albeit down from $205.6 million a year ago.

    Although surpassing revenue expectations, the cloud-based data warehousing business saw its valuation tumble 16 percent in extended trading on Wednesday. Its stock price dived from $133 apiece to $117 in after-hours trading, and today is cruising back at $127. That stumble arrived amid a general tech stock sell-off some observers said was overdue.

    Continue reading
  • Amazon investors nuke proposed ethics overhaul and say yes to $212m CEO pay
    Workplace safety, labor organizing, sustainability and, um, wage 'fairness' all struck down in vote

    Amazon CEO Andy Jassy's first shareholder meeting was a rousing success for Amazon leadership and Jassy's bank account. But for activist investors intent on making Amazon more open and transparent, it was nothing short of a disaster.

    While actual voting results haven't been released yet, Amazon general counsel David Zapolsky told Reuters that stock owners voted down fifteen shareholder resolutions addressing topics including workplace safety, labor organizing, sustainability, and pay fairness. Amazon's board recommended voting no on all of the proposals.

    Jassy and the board scored additional victories in the form of shareholder approval for board appointments, executive compensation and a 20-for-1 stock split. Jassy's executive compensation package, which is tied to Amazon stock price and mostly delivered as stock awards over a multi-year period, was $212 million in 2021. 

    Continue reading
  • Confirmed: Broadcom, VMware agree to $61b merger
    Unless anyone out there can make a better offer. Oh, Elon?

    Broadcom has confirmed it intends to acquire VMware in a deal that looks set to be worth $61 billion, if it goes ahead: the agreement provides for a “go-shop” provision under which the virtualization giant may solicit alternative offers.

    Rumors of the proposed merger emerged earlier this week, amid much speculation, but neither of the companies was prepared to comment on the deal before today, when it was disclosed that the boards of directors of both organizations have unanimously approved the agreement.

    Michael Dell and Silver Lake investors, which own just over half of the outstanding shares in VMware between both, have apparently signed support agreements to vote in favor of the transaction, so long as the VMware board continues to recommend the proposed transaction with chip designer Broadcom.

    Continue reading

Biting the hand that feeds IT © 1998–2022