Ground Systems Testing
Norm Strang
Testing procedures are seldom fully foolproof, but an independent review can help identify and correct potential sources of trouble.
This is an example of what can happen when test procedures are not followed properly or were not written correctly. This payload accident probably cost millions of dollars in equipment replacement and program schedule slips. The accident was caused by improper support of the payload during handling operations. (Photo courtesy of NASA) |
Ground systems testing covers many different aspects of the total ground operations, including areas such as launch facilities, power supplies and generators, fire protection, fluid storage and transfer, air conditioning, payload facilities, fixed and mobile tracking stations, communications, and vehicle transport. These testing operations begin with component testing and end with integration and testing of the complete space system. The goal is to ensure not only that systems function properly, but that they pose no safety hazard for workers in the vicinity.
An important function that Aerospace performs for the government is the review of ground facility test plans and procedures. These documents, generated by the contractors, must be composed with rigorous attention to detail. The independent review helps verify that the tests will be performed as intended and will not damage equipment or present a safety risk. In the past, these reviews have revealed major problems that could be corrected before they caused a mission failure. It has also happened that test plans, implemented without adequate review, contained problems that were only revealed by a subsequent mission anomaly or failure.
Problems Caught in Time
Human error is a major source of problems in ground systems testing. Errors can arise when procedures are not detailed enough, not interpreted properly, or not performed correctly. An incomplete set of instructions, when followed literally, can lead to serious consequences. For example, Aerospace reviewed a procedure for proof testing booster propellant tanks. The procedure entailed filling the tanks with water, pressurizing them, and subsequently draining them. The procedure was written so that the drainage valve could be opened before the tank's pressurizing gas valve was opened. This would have caused a negative pressure in the tank as the water drained out, which would have damaged the tank and could have caused a failure in flight. The procedure had to be rewritten to incorporate more detailed operations, warnings, and a final quality-assurance check.
This diagram provides an overview of a test and integration master plan, showing the relationship among testing, integration, verification, and validation. Qualification testing provides proof that the delivered item can meet the specified environments with margin for uncertainties. Low-level testing verifies lower level requirements and provides confidence for going to the next level of integration. Acceptance testing provides confidence that there are no workmanship issues. |
Similarly, a procedure for testing a solid rocket motor was written in such a way that the equipment used to lift the motor could be lowered onto the motor case at a speed and inclination that would have caused an impact severe enough to damage it. If that had happened, the motor probably would have failed after ignition.
A functional test procedure for a space booster was written so that the engine propellant valves would be cycled at the same time, with the cover on the rocket engine nozzle. Because of the difference in size of the booster propellant tanks, this allowed the smaller fuel tank to have a higher air pressure than the liquid-oxygen tank, as a result of the ambient temperature. Therefore, when the valves were opened, air from the fuel tank flowed through the engine injector, carrying hydrocarbon residue from previous static firing operations back into the liquid-oxygen tank. This could have caused a liquid-oxygen/hydrocarbon explosion during flight had it not been found—accidentally—during some special checkout operations. This problem illustrates the importance of considering the total system during test procedure development. In this case, the nozzle cover should have been removed.
Sometimes, problems arise through a lack of realism in the testing process, or through an incomplete assessment of the working environment. For example, the acceptance test procedure for a new ground station in Thule, Greenland, was written such that the hydraulic fluid lines to the antenna were not tested at actual working pressures and temperatures. During a walk-through inspection later in the certification process, Aerospace noted this deficiency. It turned out that these lines had to be reworked to add additional expansion joints. This could have caused a mission delay, had the Aerospace engineers not been involved.
Similarly, during the review of an acceptance test procedure for a new ground station, Aerospace noted that the radome foundation and the antenna foundation were joined together, without an isolation joint. A review of the antenna mission requirements showed a clear specification for high signal resolution. Without an isolation joint, the wind-induced vibration from the radome would be transmitted through the foundations, allowing the antenna to vibrate. This would adversely affect signal resolution. As a result, the foundations had to be redesigned to incorporate an isolation joint.
Even ancillary safety systems must be tested for safety. While reviewing another ground station acceptance test plan, Aerospace determined that the fire suppression system in the computer areas could be inadvertently activated by a wastebasket fire or by minor events such as a sprinkler-head malfunction or a spurious activation signal. This would dump water onto the computers, causing major damage and potential loss of mission. The problem was solved by installing an emergency shutoff switch to be manually activated when these or any similar problems occurred.
Problems Discovered Too Late
The mobile launch platform is a major part of the "clean launchpad" concept for the EELV. It contains all of the systems needed to launch the booster. All systems are checked out and tested off the pad, and then moved to the pad, where propellants and gases are loaded. All systems must be tested to ensure they work properly the first time or the mission will, at the very least, be delayed. The major ground systems involved are the umbilical systems, vehicle hold-down system, air conditioning, electrical power, electronic sequencing, trains, and tracks. (Photo courtesy of Lockheed Martin) |
Over the years, missions have failed because of major problems that were overlooked because of inadequate testing. A general lack of testing rigor, complacency in part qualification, and failure to consider important systems as a whole have all contributed to past mission failures. For example, cascading relays shut down a ground station during a high-priority mission. The problem was triggered when a short circuit caused the electronic relays in one system to open and dump their electrical load onto the next system, causing that system's relays to open, and so on. An analysis of the complete electrical power system would have revealed the possibility for this type of failure to occur. This would have triggered a system requirement to test for this type of failure. Proper testing would have required that the relays be subjected to the highest power level possible. The system could then be modified to protect against any problems observed. Had proper testing been performed during this system's development and acceptance testing, this design problem would have been found long before it caused the loss of a mission.
In another instance, a hydraulic system failure caused a booster to go out of control, leading to its destruction. Aerospace helped conduct the failure investigation, which traced the problem to the hydraulic pump pistons, which got stuck in the cylinders because they were too large. These pumps had been used successfully on many other missions. The failure investigation determined that new personnel in the machining facility introduced a drawing error into the machining operation, and as a result, the pistons were made too large. If the acceptance test procedure for the pump had been written to require testing at maximum operating conditions, instead of at much lower time and cycle rates, the problem would most likely have been found, and the launch would have been saved.
Ground systems are highly complex, and modifications to any one component, no matter how simple, can have a profound impact on all the others. Testing procedures must therefore be sufficiently thorough to account for any component changes—but this is not always the case. For example, the ground systems fuel loading line to a booster vehicle was not tested after it had been modified. As a result, an area in the line that trapped air was not discovered. Had the test procedure included checking the volume of fuel in the fuel line, it would have shown that an air bubble was displacing fuel. This caused the vehicle to be loaded 135 kilograms light, which caused a premature shutdown of the booster during flight and loss of the mission.
During a commercial launch, the first and second stages of a space booster failed to separate. The failure was traced to the vehicle separation slide mechanism, which had evidently seized up. An investigation revealed that during testing of the vehicle integration, a problem was noted in this separation system. The quality-assurance engineer and the test engineer reviewed the problem at the time, going back through the checkout procedure step by step. The test engineer determined that per the procedure, the separation system was acceptable. The quality-assurance representative disagreed, but was overruled. This example clearly illustrates the need for a formal system that requires agreement between the test engineer and the quality-assurance inspector and prohibits one from overriding the other. During the flight failure investigation, Aerospace determined that the test procedure used to check the slide mechanism did not have enough detail, and as a result, it was interpreted incorrectly.
Problems with rocket boosters can arise during integration or final preparations at the launch site, and ground checks provide the last opportunity to find and correct them. Skimping on these final checks can have disastrous consequences. For example, in an effort to reduce cost and weight of a booster vehicle, a transducer was removed from the booster system used for monitoring pressurization of the liquid-oxygen tank. The transducer was originally located downstream from the heat exchanger, which generated gaseous oxygen, in a bend in the line with a flex hose connected to it. The transducer was removed, and an elbow was installed in its place. No testing was performed. During the first launch with this configuration, the mission was lost because the flex hose failed after experiencing resonance, which allowed the pressure in the liquid-oxygen tank to decay. Had this change been properly tested, the problem would have been found, and the mission saved.
Overtesting
Space Launch Complex 6 in Vandenberg, California, was built to launch the space shuttle but was never used for that purpose. It has since been modified to launch the Delta IV EELV. Testing of this launch pad is complicated by the many systems that were modified from the original design or were abandoned in place. This increases the chance that the checkout procedures and testing operations will be misinterpreted or incorrectly performed. Special precautions must therefore be taken, which increases the testing burden. ((Image courtesy of The Boeing Company) |
It's easy to conclude that ground systems can never be tested enough—but in fact, overtesting can be as big a problem as undertesting. Testing is an invasion of the system, and provides new opportunities for injecting human error. The more a system is tested, the more likely it is that new errors will be introduced. There's an old saying, "If it ain't broke, don't fix it," and taking apart a good system to do more testing ignores this simple wisdom. Overtesting also adds cost to the program by funneling time and resources into activities that may no longer be providing benefit. Therefore, only needed testing should be performed based on a detailed understanding of how the system functions. The amount of testing required should be based on past experience with similar designs, materials, and levels of complexity. In this case, the years of experience that Aerospace has gained through involvement with diverse space programs can help determine the proper amount and the proper type of testing.
Conclusion
Ground systems operations depend on total systems engineering to ensure proper design and development. This includes knowing that the design is correct, proper materials were used, proper manufacturing was performed, proper assembly operations were conducted, and proper testing was applied. Each of these steps requires successful testing before proceeding to the next step. If not, system failure is all but inevitable.
Ground systems represent the largest overall cost for most space programs. However, testing of ground systems does not always get the same visibility as vehicle testing, for example. This is a major concern because problems with ground systems are just as likely to cause a mission failure as are vehicle problems. Also, ground systems tests are more prone to human error, ranging from a lack of detail in writing the test plan to a failure to understand and implement the testing protocols. Test plans must be meticulously written to prevent errors, to protect personnel, and to ensure a high level of confidence in the results. This is best achieved through a formal system designed to ensure that the technicians are well trained, the procedures are well written and approved by qualified reviewers, and the testing operations and results are accepted in a formal approval process. An independent review, such as that performed by Aerospace, is an important part of this process.
