The picture perfect docking of the SpaceX Dragon capsule to the International Space Station (ISS) on March 3 and the triumphant ocean splashdown last week on March 26 nearly weren’t to be – and it all goes back to a microscopic manufacturing mistake in the oxidizer tank check valves that no one noticed long before the vessel ever took flight.
Barely 11 minutes after I witnessed the spectacular March 1 blastoff of the Dragon atop the SpaceX Falcon 9 rocket from Cape Canaveral, Florida, everyone’s glee suddenly turned to disbelief and gloom with the alarming news from SpaceX Mission Control that contact had been lost.
I asked SpaceX CEO and founder Elon Musk to explain what caused the failure and how they saved the drifting, uncontrolled Dragon capsule from doom – just in the nick of time.
Applying the space version of the Heimlich maneuver turned out to be the key. But if you can’t talk to the patient – all is lost.
Right after spacecraft separation in low Earth orbit , a sudden and unexpected failure of the Dragon’s critical thrust pods had prevented three out of four from initializing and firing. The oxidizer pressure was low in three tanks. And the propulsion system is required to orient the craft for two way communication and to propel the Dragon to the orbiting lab complex.
So at first the outlook for the $133 million Dragon CRS-2 cargo resupply mission to the ISS appeared dire.
Then, SpaceX engineers and the U.S Air Force sprang into action and staged an amazing turnaround.
“The problem was a very tiny change to the check valves that serve the oxidizer tanks on Dragon.” Musk told Universe Today
“Three of the check valves were actually different from the prior check valves that had flown – in a very tiny way. Because of the tiny change they got stuck.”
SpaceX engineers worked frantically to troubleshoot the thruster issues in an urgent bid to overcome the serious glitch and bring the crucial propulsion systems back on line.
“What we did was we were able to write some new software in real time and upload that to Dragon to build pressure upstream of the check valves and then released that pressure- to give it a kind of a kick,” Musk told me at a NASA media briefing.
“For the spacecraft you could call it kind of a Heimlich maneuver. Basically that got the valves unstuck and then they worked well”
“But we had difficulty communicating with the spacecraft because it was in free drift in orbit.”
“So we worked closely with the Air Force to get higher intensity, more powerful dishes to communicate with the spacecraft and upload the software to do the Heimlich pressure maneuver.”
Just how concerned was Musk?
“Yes, definitely it was a worrying time,” Musk elaborated.
“It was a little frightening,” Musk had said right after the March 1 launch.
Later in the briefing Musk explained that there had been a small design change to the check valves by the supplier.
“The supplier had made mistakes that we didn’t catch,” said Musk. “You would need a magnifying glass to see the difference.”
SpaceX had run the new check valves through a series of low pressurization systems tests and they worked well and didn’t get stuck. But SpaceX had failed to run the functional tests at higher pressures.
“We’ll make sure we don’t repeat that error in the future,” Musk stated.
Musk added that SpaceX will revert to the old check valves and run tests to make sure this failure doesn’t happen again.
SpaceX, along with Orbital Sciences Corp, are both partnered with NASA’s Commercial Resupply Services program to replace the cargo up mass capability the US lost following the retirement of NASA’s space shuttle orbiters in 2011.
Orbital’s Antares rocket could blast off on its first test mission as early as April 17.
Of course the Dragon CRS-2 flight isn’t the first inflight space emergency, and surely won’t be the last either.
So, for some additional perspective on the history of reacting to unexpected emergencies in space on both human spaceflight and robotic science probes, Universe Today contacted noted space historian Roger Launius, of the Smithsonian National Air & Space Museum (NASM).
Roger provided these insights to Universe Today editor Nancy Atkinson – included here:
“There are many instances in the history of spaceflight in which the mission had difficulties that were overcome and it proved successful,” said Launius.
“Let’s start with Hubble Space Telescope which had a spherical aberration on its mirror and the first reports in 1990 were that it would be a total loss, but the engineers found workarounds that allowed it to be successful even before the December 1993 servicing mission by a shuttle crew that really turned it into a superb scientific instrument.”
“Then what about Galileo, the Jupiter probe, which had a problem with its high gain antenna. It never did fully deploy but the engineers found ways to overcome that problem with the communication system and the spacecraft turned into a stunning success.”
“If you want to feature human spaceflight let’s start with the 1999 shuttle flight with Eileen Collins as commander that had a shutdown of the SSMEs prematurely and it failed to reach its optimum orbit. It still completed virtually all of the mission requirements.”
“That says nothing about Apollo 13,… I could go on and on. In virtually every mission there has been something potentially damaging to the mission that has happened. Mostly the folks working the mission have planned for contingencies and implement them and the public rarely hears about it as it looks from the outside like a flawless operation.”
“Bottom line, the recovery of the Dragon capsule was not all that amazing. It was engineers in the space business doing what they do best,” said Launius.
…………….
Learn more about SpaceX, Antares, Curiosity and NASA missions at Ken’s upcoming lecture presentations:
April 20/21 : “Curiosity and the Search for Life on Mars – (in 3-D)”. Plus Orion, SpaceX, Antares, the Space Shuttle and more! NEAF Astronomy Forum, Suffern, NY
April 28: “Curiosity and the Search for Life on Mars – (in 3-D)”. Plus the Space Shuttle, SpaceX, Antares, Orion and more. Washington Crossing State Park, Titusville, NJ, 130 PM
SpaceX Falcon 9 rocket and Dragon capsule poised to blast off from Cape Canaveral Air Force Station, Florida on a commercial resupply mission to the ISS. Credit: Ken Kremer/www.kenkremer.com
In a strange way, I am continually more impressedwith SpaceX every time they _fail_ (okay, that’s a bit strong: let’s downgrade that to simply… when something goes unperfectly). They are consistently able to get the right information to the right computers/people within the right timeline in order to make the right decision. Compare that process, for example, to NASA and the Columbia space shuttle disaster….
Did you not read about the successful NASA examples? One day SpaceX is going to have its own fatal accident. This isn’t to say anything negative about SpaceX because its only inevitable in the space business
exactly right. However, I also think that SpaceX will handle it better. The reason is that they will not have CONgress threatening them with pulling of funding etc. SpaceX will continue to have lots of launches due to private space.
NASA really has been between a rock and a hard place with CONgress squeezing them with it. That is in no small way why NASA WANTS AND NEEDS cargo and human launches to go to private space.
NASA should only be focused on going BEO to the moon and mars.
While I am a major supporter of SpaceX, It always bothers me when I see others that pull up all sorts of BS about NASA without grasping what they are saying.
The fact is, that NASA, does a great jobs with information flow considering how immense there was on the shuttle. The fact is, that the falcon and dragon has far more sensors than the shuttle and all of them feed to computers that do the real work. Sadly, the shuttle was a combination of electronic and manual control and monitor. As such, it took so much more information flow CONSTANTLY, that any little issue would be ignored after a while. With SpaceX, it is almost all electronic control and monitor.
BTW, the ones to blame for the shuttle issues was ultimately, CONgress, not NASA.
The space shuttle was a stupid idea and NASA came up with it.
Actually, the shuttle as proposed by NASA was actually brilliant. The problem is that Nixon would not fund it and pushed it to replace the Saturn V.
What? Nixon would not fund it yet pushed it?
It was built. What more money was needed? It was fundamentally inefficient.
You are lifting a massive airplane into space. Think about the payload you could manage to get into space with those massive non-reusable rockets you are using to lift it there. All fit snug in a nice compact perfectly aerodynamic rocket.
NASA wanted a ‘spaceplane’. Basically, a plane that would lift a vehicle to about 100K’ at a speed of around mach 6 or more. From there, the ‘plane’ would seperate and the upper part would then ignite a SMALL rocket engine (most likely H2/LOX) that would carry the craft to LEO. This was IDEAL for taking say 7 ppl OR a small cargo to LEO.
BUT, that would involve money for the R&D.
NIXON killed it. He did not want us to have a CHEAP space program. that would mean going back to the moon which he did not want to spend money on.
So, he pushed NASA and USAF to work together on a new cheap to build craft that would work for both.
Sadly, it was none of the above.
You are saying some odd things. He didn’t want a cheap program because it would mean going to the moon? Why not have a cheap program and NOT go to the moon? (Not that I would prefer that necessarily)
Does your next sentence imply that he had them work on a plan that would take them back to the moon?
You know, we still don’t have a spaceplane today in spite of much research being done on it. I think you are right that Nixon didn’t give them as much money as was needed, but I don’t think he required of them a spaceplane. As you state “NASA wanted a spaceplane”. If reality dictates you won’t get enough money to do it right, you cannot pardon them for doing it anyway.
When recently looking into this I read that NASA also chose an inferior developer of solid rocket boosters due to conflicts of interest. You can only blame NASA for that.
Are you implying that NASA wanted to keep the Saturn V for longer?
Not implying anything. NASA fully expected to have a SHLV. They testified to it. Saturn V was NOT designed for a short set of moon shots. It was started in the late 50’s before kennedy was to send us to the moon.
Sadly, Nixon and ‘conservatives’ of that time killed it.
But what was the difference and how did they find it? How did normal testing not catch something that frantic backtracking did?
Maybe you need a microscope to *see* the difference, but it sounds like there was another sensor or diagnostic which could and did tell, perhaps with unexpected sensitivity.
They knew that they had gotten a new valve from their supplier. They slipped in that they didn’t run it through a full range of pressures during their simulations. In flight the three thrusters that had the new valves failed so it wasn’t that hard to figure out what happened. The good news here is it seems that the engineers have direct access to the mission command team and they communicate efficiently. This was clearly not the case with the 2 shuttle disasters where there was a clear disconnect between the engineers and mission control regarding hazards during launch and their risk to the mission.
I do agree, the report from SpaceX is very fuzzy and especially not detailed.
We cannot test that claim if that was indeed the real cause.
I am glad that SpaceX was willing to reveal the nature of the problem in an honest fashion. I think ultimately that approach enhances confidence in their company.
You are so correct. I know that on the Atlas V, it took over 3 flights to have that fully shook down. They had issues crop up that was solved, but was not spoken about.
SpaceX is the exact opposite. They continue to push publicity about who they are, and take their lumps when they come.
Best be to honest. A fib, a half lie will bite you when you least expect it. Good for them.
Nice report , also your pic. !!
Visions of Dragon’s grandson surrounded by doughnut shaped Bigelow modules and storage tanks. The tanks stacked cylindrically with propulsion module and bulkhead attached aft. This craft is spun up for synthetic gravity and solar sail deployment. Mars in 100 days or BUST!
great write-up.
One of the things that bugged me was that so many ppl gave all the credit to SpaceX, when it was obvious that it was multiple groups working together.
And that is what is shown here. They worked together to solve this since SpaceX, NASA, and DOD all have vested interests in this.