A Program For Success: The Aerospace Role in Software Assurance

John Cantrell, Suellen Eslinger, and Robert Duvall

Aerospace works to ensure that mission assurance is a component of every phase of software development.

Software size and complexity are on the increase and have been since the first computer programs were developed in the 1950s. Unfortunately this development has been accompanied by an increased chance of error. More and more often, failures in space can be traced to software problems. The first Ariane 5 launch ended in failure because of reuse of Ariane 4 software combined with specification and design errors and inadequate testing. The Mars Climate Orbiter was lost because different units were used for parameters interpreted by software for different parts of the system. The most likely cause of the failure of the Mars Polar Lander appears to be inadequate specification of known physical behavior. These accidents are discussed by Nancy Leveson of the Massachusetts Institute of Technology in her 2004 article in Journal of Spacecraft and Rockets, "The Role of Software in Spacecraft Accidents."

These failures in space, although attributed to software, have broader implications for mission assurance. They can be attributed to many factors ranging from inadequate definition or communication of requirements to the failure to test for correct implementation of requirements. Mission assurance for software must cover the entire software life cycle. It must play a role in requirements definition; requirement flow down to the subsystems; and software design, coding, integration, and testing. Software mission assurance is not a quality that can be bolted on or tested just before launch. Rather, it must be engineered into the entire system from inception.

The Aerospace Corporation works to ensure that mission assurance is a component of every phase of software development. The corporation's work in software standards helps put customer contracts on a good foundation for software development. Monitoring the software development effort over the entire life cycle raises the customer's confidence that the contractor is adhering to recognized good development processes. Finally, by scrutinizing the thorough testing of the software, from unit testing, through integration up to qualification testing, and even validation and verification efforts, Aerospace can alert both the contractor and the customer of national space systems to problems early, when correcting them is less expensive.

The Role of Software Engineering

Good software engineering practices provide one pillar supporting mission assurance. Development of quality software includes the following essential fundamentals.

Have a process and use it. A documented well-understood software development process provides from the outset an understanding of the goal of the development effort and prevents the effort from becoming interminable.
Strive for high quality. Managers and software leads should stress the necessity for software that is reliable and safe.
Involve the customer. Customer feedback is not just important in the world of business.
Implement management fundamentals. Personnel and equipment resources must be carefully planned and continually monitored and managed. Metrics to help monitor and manage software quality and productivity should be collected.
Implement software engineering fundamentals. This starts with requirements management and moves through design, construction, and testing of the software. Consistent application over the entire life cycle is key.
Use quality assurance fundamentals. Technical reviews must be a regular part of development. Peer reviews tend to locate defects earlier than testing and identify different types of defects (e.g., requirements and design flaws). Testing and reviews should complement one another.

Certain steps should be included in whatever process the development organization chooses to use. These include requirements definition for the software, architectural design, detailed design, coding, and finally testing. These steps may or may not overlap, depending on the design methodology. The process should describe what is necessary in each step. A systems engineer present at the requirements definition meetings should provide a broader perspective than that coming from software engineers alone. A test engineer present during the requirements definition meetings should lead to requirements that can be tested.

While it is possible for the steps in the process to overlap, any step that is missed or any shortcut taken increases mission risk. Not having a test engineer during requirements definition may result in requirements that are difficult to test. This may necessitate a requirements change, the acquisition of expensive test equipment, or software changes because of ambiguities. While it is not a foregone conclusion that these outcomes will occur, the presence of a test engineer may well prove to be a cheap insurance policy.

Technical review of the code and design is another area that is sometimes overlooked. When programmers review each other's design and code, everyone strives to do a better job. In addition, changes made in one software module as a result of a review usually are implemented immediately in subsequent modules, obviating the need for the same change repeated again and again. Such practices contribute to programmer education and to overall code quality.

Setting the Standards

Aerospace cannot dictate to contractors how they should handle their software development efforts. However, Aerospace can gain insight into the processes that the contractor uses and, based on experience, make judgments about the efficacy of these processes. All peer reviews, for example, are not the same, even within a single contractor organization.

Aerospace's long experience guides its efforts to monitor the contractor's performance. Are the processes followed by the contractor in line with its promises (software development plan, software test plan)? Do the plans make sense in the program environment? How could they be improved within schedule and budget constraints? These are just some of the questions that Aerospace answers in striving to provide mission assurance.

When a dramatic increase in on-orbit anomalies and space-vehicle failures occurred in the late 1990s and early 2000s, anomaly investigations and major cross-program studies cited a foremost root cause as the lack of adequate processes by the development contractors, especially processes for testing and other activities related to mission assurance. Software was affected as well as hardware. With the removal of software-related military standards from contracts, military space-system contracts mandated no specific contractual requirements for software testing and other critical activities related to software mission assurance, such as peer reviews of software work products. Eliminating these requirements led directly to a dramatic increase in on-orbit anomalies and mission degradations caused by software.

An actual launch vehicle flight computer is a rugged closed system well suited for flight, but not ideal for validation of flight software in the laboratory. The highly integrated design makes the attachment of internal electrical and logical probes difficult.

Aerospace has been a leader in defining contractual requirements driven by mission assurance (see sidebar, The Chief Software Engineering Advisory Council). In 2003, the Space and Missile Systems Center/National Reconnaissance Office (SMC/NRO) Mission Assurance Improvement Task Force was initiated, revitalizing emphasis on contractual standards. The principal software-related product of this effort was an update of the military standard for software development (MIL-STD-498), which had been canceled during the acquisition reform initiatives of the mid-1990s. The team modified MIL-STD-498 to bring the standard up to date and to add requirements related to software mission assurance. The updated standard is documented in an Aerospace technical operating report, "Software Development Standard for Space Systems."

The additional requirements related to mission assurance are based on experience with multiple space programs and on well-researched and documented results from the software engineering literature. These requirements address peer reviews; traceability; assurance of critical requirements for dependability, reliability, maintainability and availability, and key performance parameters; software unit testing and software unit integration testing; hardware and software integration testing; and software qualification testing, quality assurance, and metrics. Although titled "Software Development Standard for Space Systems," the updated software standard actually applies to any software-development effort.

From a mission assurance perspective, the importance of building quality into the software prior to testing (e.g., using such techniques as peer reviews, software quality assurance reviews, software product evaluations, and joint technical reviews) is paramount. A robust software test program, however, is the last line of defense against latent defects in the software that could cause mission loss or degradation, and thus is critically important to any successful space program.

The robustness of a test program is determined by its depth and breadth. For depth, the software standard requires multiple levels of software testing: unit testing, integration testing (both among software units and among hardware and software), and qualification testing. For breadth, specific test coverage criteria are required by the standard for each level of software testing. Software unit testing must successfully test all statements and branches in the code, all error and exception handling, all unit interfaces (including limits and boundary conditions), and all algorithms. Software integration testing must successfully test all interfaces among the software and hardware (including limits and boundary conditions), end-to-end functional and performance requirements under conditions reflecting operations, stress testing including worse-case scenarios, fault detection and recovery handling, and start-up, termination, and restart. Software qualification testing must successfully verify all software and interface requirements—whether satisfied by new, reuse, or commercial off-the-shelf (COTS) software—on the target processing hardware using the actual interfaces where possible and high-fidelity simulators where not possible. Based on lessons learned from space-system failures caused by software, the software-development standard was updated to also require regression testing of affected unit, integration, and qualification test cases for any modifications to previously tested software. In addition, the standard specifies minimum requirements for testing of COTS and reuse software.

The SMC specifications and standards effort (part of the SMC systems engineering revitalization initiative) has been instrumental in updating and reintroducing other military specifications and standards important to space-system mission success. One of the most important of these is the standard for space-vehicle test requirements (MIL-STD-1540), which has been updated for enhanced mission assurance (documented in an Aerospace technical report, "Test Requirements for Launch, Upper Stage and Space Vehicles"). Aerospace was responsible for getting software test requirements included in this standard for the first time in its history. This standard now includes requirements for a robust software test program appropriate to onboard software. In addition to the test requirements in the new software standard, the update to MIL-STD-1540 includes requirements for testing the software during thermal vacuum tests where the processor speed can be affected by the environmental temperature conditions. Another software test requirement driven by lessons learned with actual space-system failures is the requirement to regression test the software when the operational values of the onboard constants database are loaded. The updated software development standard and the updated space-vehicle test requirements standard are now being cited as compliance standards on new SMC and NRO contracts.

Software Qualification and Spacecraft Bus Testing

Qualification testing for software in national space systems is the verification that the software correctly implements its requirements. While qualification testing cannot guarantee mission success by itself, it improves overall confidence that the contractor understood the software requirements and implemented them, and that the software will perform as designed once on orbit. It is another building block in the foundation of confidence supporting mission assurance.

Qualification testing activities usually occur at the end of the software development cycle. If the final deadline (launch) of a system is fixed, which is frequently the case, and the software development effort underestimated the time required to implement the software, pushing back delivery of the software to the testing phase, the testing effort is compressed. Consequently, the launch may take place before all software has undergone qualification testing. What happens now?

In terms of the payload, Aerospace has identified key functions that are more essential to mission assurance than others. While all payload functions are clearly important, these key functions have been identified as such because software failure in any one of them could cause loss of the payload. An Aerospace technical report—"Recommended Process for Qualification Testing of Payload Software Prior to Use on Orbit"—identifies core functions that must undergo complete qualification testing prior to launch. These functions are supplemented by ancillary functions that could be qualified on the ground using simulators before being uploaded in orbit. The report does make clear that all flight software should be qualified before launch, but real situations often allow less-than-ideal actions.

The various payload functions are classified as core—capabilities that require qualification testing before launch—and basic—capabilities that may be qualified and uploaded to the payload after launch. Booting up, for example, is a core function that describes the ability of the payload to start after the application of power and to enter a known safe state defined by the payload designers. The safe state might mean any state from the payload waiting for a ground command to the payload in full-blown operation. What is important is that the payload is able to enter this state after power is applied. If an error occurs while booting up, the payload may be lost. For this reason, this function needs to be tested as much as possible before launch.

Core functions include booting up, spacecraft communications, system upload, reporting the status of hardware and software, and "safing" the payload. Basic functions are hardware control and payload data processing.

Spacecraft communications assumes the payload does not communicate directly with the ground but rather communicates by way of the spacecraft. If this capability does not operate properly, the payload cannot offload its mission data to the ground and the ground cannot command the payload. In this case, the payload capability would be lost.

If perfect software is loaded at launch, it is possible that the system upload capability may never be used, but it may be necessary to exercise this function at some future date—for example, if a new software system upload is needed. If qualification testing and the preceding testing efforts were not thorough enough, a problem may leave the payload incapable of either starting the new system or returning to the previous version of the software.

The hardware and software status reporting capability is considered a core function mainly because of its value in the event of an anomaly. If everything is working correctly with the payload, this function will only reinforce what is already known—that everything is working correctly. However, when things go wrong, this may be the only way ground controllers are able to analyze the root cause of the anomaly and devise a correction. This capability is especially important during the early life of a payload, when ground controllers are not thoroughly familiar with its behavior.

Safing the payload is another capability that may never be exercised, but if the payload must be put into a safe state because of an anomaly, this capability will be called upon. Depending on the payload, this capability may require that the mechanical doors be closed, power be removed or reduced, or other actions be taken to protect the payload from the environment of space. In addition to allowing the payload to enter into a safe state, this capability must also allow it to exit this safe state—most likely back to the boot-up function. This entire path, from anomaly to safing to recovery, must undergo extensive testing before launch to provide maximum confidence that it will perform its function after launch.

Hardware control and payload data processing, on the other hand, are considered basic, or ancillary, functions. Proper testing of the actual flight hardware involved in these actions usually involves the use of test (i.e., not flight) software. Such testing may not have been at the qualification level, but it does show that the hardware and data processing can be controlled properly with software. If the space vehicle must be launched before such software capabilities are qualification tested, this software can be omitted from the launch configuration of the software and uploaded later. In this case, mission availability is negatively affected because the payload will not function as designed until all software components are operating onboard. Because all core functions are working as designed, the hardware control and payload data-processing capabilities can be tested thoroughly on the ground and uploaded and tested in flight, allowing full-scale operations to be started. This saves schedule time on one side of the launch at the cost of schedule on the other side.

The list of core and basic functions for spacecraft bus capabilities is longer for this software-intensive system. Capabilities considered core for the payload must also be considered core for the corresponding capabilities on the space vehicle. Communication, for example, is considered core for both payload and space vehicle because it is essential for them to communicate with each other. Another example is booting up the spacecraft, which performs much the same actions as booting up the payload. Other spacecraft core functions analogous to those of the payload are system upload, hardware/software status reporting, and safing the spacecraft.

Several additional capabilities are considered core for the spacecraft: the executive/operating system, whether a commercial product or purpose-built by the contractor; guidance, navigation, and control; thermal control; and electrical power control. Any failure of these systems because of software puts the entire mission in jeopardy. They must be exercised as thoroughly as possible before launch to increase confidence that this software and the accompanying hardware will work together to accomplish their subsystem tasks. Another core function is on-orbit deployments and beginning-of-life testing.

This hardware-in-the-loop simulation was developed at Aerospace for the Atlas V. The single board computers are built with the same processors, bus controller chipsets, and other electronic components that are used in the flight unit. These boards, however, plug into an equipment rack with standardized data-bus connections that allow easy use of logic analyzer probes to measure signals and timing. The boards also have special probe pads that allow direct access to the central processing unit (CPU) buses. This open design allows complete access to the flight software behavior with a resolution of a single CPU clock cycle. Aerospace currently maintains hardware-in-the-loop simulations for the Delta II, Delta IV, and Atlas V launch vehicles.

Command execution and onboard command storage must also be subjected before launch to as much testing as possible, usually involving all possible commands. If the spacecraft cannot execute commands issued from the ground, mission goals would not be achieved. For this reason, ground communications are also considered core. Onboard command storage is usually provided to store commands for future execution. These may be certain command sequences, such as the actions needed to safe the spacecraft that must be activated quickly in the event of a fault, or they may be sequences that allow ground controllers to upload one command while the spacecraft actually performs a large number of commands. Often, these command sequences are used at the beginning and end of life for the spacecraft—the function of storing commands and then executing them at the appropriate time is vital to the health of the spacecraft and the mission.

A basic function that could be considered core involves end-of-life maneuvers, which have gained importance in recent years. These end-of-life maneuvers may be a controlled deorbit of the spacecraft, moving it to a higher orbit where it will not jeopardize other spacecraft or leaving it where it is, if that is an option. In this particular case, though, any hardware used by these maneuvers will have been tested, and as these maneuvers are not scheduled for years after launch, they only become critical at that time. There are circumstances in which these maneuvers may become needed far earlier than planned, such as an anomaly with the launch vehicle that puts the spacecraft in the incorrect orbit. This situation would drastically shorten the testing schedule for such software. This capability is still considered basic because it can be tested at relative leisure on the ground before being uploaded to the space vehicle in most cases. Other basic functions are onboard data storage and retrieval and onboard data processing.

Flight Software Validation

Although it is never a good idea to launch a mission with software that has not undergone complete qualification testing, Aerospace has provided a path to launch that offers more flexibility when reality prevents optimal testing.

Aerospace develops and maintains hardware-in-the-loop (HIL) simulations for the validation of flight software for both satellites and launch vehicles. A hardware-in-the-loop simulation is built from a combination of flight-equivalent avionics that can execute the same flight code image and mission data that will be loaded on the real system for flight. This "test-like-you-fly" approach gives the most realistic evaluation of how the system software will perform during flight. In most cases, the hardware-in-the-loop simulation runs in real time, meaning that the flight software is being executed at the same speed as it would on the real system.

The simulation emulates all input and output (I/O) interfaces that are visible to the processing hardware on which the flight software runs. The I/O interfaces are driven by high-fidelity simulations of all vehicle subsystem hardware, as well as vehicle dynamics and flight environment models. These are initially developed in the form of a "scientific simulation"—a simulation done completely in software that mathematically simulates the subject—which runs in a standard workstation environment. The models are developed and tested by running a version of the vehicle flight code that can execute on the workstation. Once the fidelity of the vehicle models has been validated, they are moved to the hardware-in-the-loop simulation.

Launch vehicle scientific simulations are also used for day-of-launch activities. The simulation executes the flight software to generate real-time predictions of launch-vehicle behavior during atmospheric flight on launch day. Wind data measured by weather balloons at the launch site are incorporated into the simulation to allow for up-to-date predictions. Structural loads and engine deflections are found from these real-time predictions and compared with launch-day limits. This capability allows Aerospace to make a go/no-go launch recommendation based on flight-software performance in the face of actual conditions at the launch site.

Aerospace also develops and maintains hardware-in-the-loop simulations for satellites. Independent analysis is carried out by developing models of the spacecraft dynamics, including all deployables (solar arrays, antennae, etc.), control sensors, and actuators. For example, the Global Positioning System Block IIR spacecraft has 13 articulated bodies, all modeled and simulated. In each hardware-in-the-loop simulation, the actual flight computer or equivalent is included. Mission assurance is enhanced by this independent analysis through the use of both scientific and hardware-in-the-loop real-time simulations that are used to validate the attitude control design, performance, and stability of the satellite. Included in this validation are the control laws that are implemented in the flight code.

Independent testing with the hardware-in-the-loop simulator leads to discovery of flight-software weaknesses, which are brought to the attention of the contractor. Workarounds or design changes are used to eliminate or mitigate the weaknesses. The simulator is used to support launch training, launch support, and on-orbit anomalies if they occur. This can include validation of the contractor's flight-code software patches before they are loaded up to the spacecraft.

Aerospace continues to improve the fidelity of the vehicle dynamics and flight environment models that are used by the scientific and hardware-in-the-loop simulations. One example is a new integrated simulation that combines a 6-degrees-of-freedom, rigid-body simulation with a computational fluid-dynamics simulation. This integrated simulation allows Aerospace to analyze the interactions between fuel/oxidizer dynamics, stability control, attitude control, and other interactions with flight code. This new simulation capability is a first step in Aerospace's plans to integrate simulation models developed in house. This systems simulation environment would more fully ensure that all subsystems that interact with software/hardware avionics are fully tested.

Aerospace has made major contributions to software mission assurance for launch vehicles and continues to pursue a number of diverse efforts to maintain and enhance its leading-edge role in software mission assurance for space systems (see sidebar, The GPS Control System Transition). These include defining the contents of critical technical contract deliverables for software, tailoring space and ground reliability standards and their associated contractual deliverables to include software, using static and dynamic software analysis tools for identifying problems in software products, evaluating contractor compliance with software development and testing processes, defining software technology readiness levels and evaluating software technology readiness for programs, and performing research in software reliability and software acquisition.

Acknowledgment

The authors thank Daniel Dayton, Douglas Buettner, Michael Tanzillo, Joseph Anselmi, and Marin Panevsky for contributing to this article.