After a full day cycle delay thanks to weather, the first Block I finally had a chance to liftoff after 4th sunrise, however it was beset by a failure of its first stage booster. The lack of ignition triggered a cascade of improper commands from the Automated Flight Control System which resulted in the premature deployment of the parachute, the second stage booster igniting and the rocket being carried almost 8km downrange to crash into the waters of the Kerblantic.
Today we investigated the first stage booster, which was left lying on the launchpad, in the VAB to determine whether it was a bad ignitor or a mis-fire that triggered the failure. A bad ignitor means that the spark that should have been created to light off the solid fuel burn was not generated due to manufacturing defects. Unfortunately there is no way to test for a bad ignitor before launch – as soon as you set it off to confirm it is working it becomes useless. The most you can do is ensure that an electrical signal is reaching the ignitor by running a small charge through the wires to establish continuity but not large enough (usually) to set it off. The launch team confirmed continuity during countdown. A mis-fire means the ignitor did generate the spark meant to ignite the solid fuel but the burn either did not initiate or was stopped prematurely. In the VAB it was confirmed that the ignitor was defective, which is good as it means we don’t need a new first stage booster, just a new ignitor.
Usually with a failure like this everything would have been fine – the rocket would have remained on the launch base and we probably would have tried again just to make sure the booster was really not able to fire. However we have suffered our first logic error via the AFCS. It’s easy to clean up syntax problems that generate errors when compiled but logic errors can usually only be found during actual execution.
In this case, the problem came down to the AFCS being unable to tell the difference between a booster that hasn’t fired and a booster that has finished firing. The main piece of offending logic can be found in this code statement. After the command is sent to ignite the first stage booster the AFCS immediately begins to monitor the status code being returned by the booster’s control unit. Since the booster registered as not firing or “flamed-out”, the AFCS thought that it was the end of the first stage boost and proceeded to monitor the coast that was supposed to follow. Coast periods are primarily determined by the pitch of the rocket changing, however there is a back-up trigger that engages the next booster if vertical speed falls below 100m/s. Therefore the second stage fired immediately as the rocket fell to the ground when the rail hooks released. It stayed attached to the lower stage just shy of a second before breaking free of its decoupler and flying off downrange.
Further compounding the situation were a series of triggers immediately set off when the rocket fell from the launch rail, as that generated a negative vertical speed. This deployed the parachute, which was then cut when the rocket hit the ground. Losing both the parachute and the nosecone was bad because it meant we could no longer communicate with the AFCS as the nose cone carries the communications antenna and the parachute could have potentially saved the third stage and payload after the second stage burned out. But if the second stage burned out and we couldn’t shut down the AFCS why did the third stage not ignite?
A second after the launch, the AFCS suffered a system crash when it tried to tell the first stage to decouple itself as it was no longer attached. If the decoupler had failed just a few milliseconds later the rocket would have most likely continued to ascend through its third stage as well unless the Range Safety Officer felt it was necessary to destroy it (the onboard Flight Termination System has its own short-range antenna expressly for this purpose).
We estimate that the rocket impacted the water at speeds in excess of 250m/s, which would not have left much intact but we have still asked the Maritime Service for a dredger to come and see if its sonar can pick up anything on the sea bed that we can maybe recover.
Here is the full log of events:
The only thing the software team hasn’t been able to figure out is why the full chute deployment message was triggered twice, or even how it could have possibly been triggered twice given the trigger was not flagged to preserve itself and was nested within another trigger that only fired once. Don’t also let the “flight operations concluded” message lead you to think that the third stage would not have been triggered if the system crash hadn’t happened – the code that ends flight operations does not account for the function monitoring the second stage boost still being active and would not have removed it from the main loop.
The AFCS team has already made corrections to prevent future occurrences of booster ignition failure resulting in the rest of the rocket thinking everything is fine. They will upload new code to the repository later next week after doing extensive brainstorming on other possible failures that could trip up the code during ascent. We have yet to talk to Tiberdyne Aerospace, who contracted this mission, to see if they are interested in another launch attempt but if not the code will be used for our next Block I launch, which is now confirmed to be still on track for the end of this month.
Here is a video of everything that was detailed above from our camera atop the flagpole north of the launch pad:
Wham! Bam! See yaaaaaaaaaaa!!
