A Successful Strategy for Satellite Development and Testing

Bill Tosney and Steve Pavlica

An Aerospace study of satellite development practices has reaffirmed the needs of the traditional approach based on uniform standards and rigorous testing.

In 1986, the President's Blue Ribbon Commission on Defense Management completed an in-depth assessment of the defense acquisition process. The recommendations of this commission resulted in a series of policy reforms geared toward a "faster, better, cheaper" acquisition strategy. One of these new policies, codified in the Military Specifications and Standards Reform Program issued by the Secretary of the Air Force for Acquisition in 1995, effectively ended the use of military specifications and standards—despite arguments that these standards represented best practices compiled through decades of costly and arduous trial and error. Commercial best practices were deemed suitable alternatives, even though in the space industry they were mostly nonexistent or unproven.

In the ensuing years, the number of late-build-cycle and on-orbit failures surged. Commercial satellite programs alone had a 146-percent increase in failures between 1998 and 2002. This increase in failures, assumed by many to be caused by deficiencies in testing, alarmed the government acquisition community. As a result, Aerospace was tasked in 2002 to conduct a comprehensive assessment of satellite testing practices to answer the question: Is testing technology failing to keep pace with changes in development methodologies, manufacturing improvements, technology upgrades, and material advances?

Out of a sample of more than 450 vehicles manufactured in the United States, those developed using traditional acquisition practices show a consistently higher success rate in the first year of operations. In contrast, vehicles developed using higher-risk acquisition approaches show markedly lower success rates in the first year of operations (view larger image).

Right away, it became clear that very few recent problems could be attributed to the failure of testing technology to keep pace with development technology—most problems arise from deeper system-wide deficiencies, particularly systems engineering shortcuts. The study found that testing is resource-intensive, but it can accommodate new technologies, when properly applied. The scope of the study was, therefore, expanded to include a comprehensive assessment of satellite acquisition and development practices, a task that Aerospace was uniquely capable of performing thanks to its comprehensive database of detailed factory and orbital data going back several decades. This allowed analysts to compare and contrast important programmatic and engineering variables affected by the acquisition changes that had taken place since 1995.

A Decline in Testing Rigor

A comparison of programmatic and engineering practices affected by the acquisition changes found that much of the growth in late-build-cycle and orbital failures occur because proven mission-assurance practices were either greatly relaxed or discarded in the wake of acquisition reform. Entrusted with more autonomy, the government and industry had grown more concerned with managing near-term cost and schedule risk than long-term performance risk—and one clear way to trim cost and schedule in the near term was by shortchanging quality, particularly testing thoroughness and issue resolution during development.

For example, development times had grown as a result of increased system complexity, and the pressure to minimize schedule slip prompted some developers to reduce test perceptiveness and thoroughness of unit, subsystem, and system-level testing and to increase the use of test surrogates instead of actual flight units in system-level tests. The time to conduct a typical system-level test, for example, had declined by an average of 30 percent since 1995. This raised questions about the rigor of the functional and performance testing employed: Was it indeed perceptive enough?

Another notable decline in test rigor was seen in the area of unit or black-box thermal testing. Contractors consistently cut back on environmental stress screening at the unit level, decreasing the number of thermal cycles by as much as 50 percent to save time. The consequence, however, was an increase in unit failures after the satellites were fully assembled and subjected to the system-level thermal vacuum test (where the cost of the failure dramatically increases).

Acquisition reform in the national security space arena reduced verification rigor, as illustrated in the drop in environmental test thoroughness (view larger image).

The space industry generally recognizes that all units (black boxes) and space vehicles should be tested under environmental and performance conditions as close to flight-like as possible—a philosophy known as "test like you fly." However, complete flight-like testing is not always feasible. There are often physical factors limiting what can be done on the ground (for example, recreating zero-gravity effects, providing the star field and environment for testing the attitude control system, providing solar illumination in vacuum for fully extended solar arrays, and fitting large systems into a relatively confined vacuum chamber). But even in light of these acknowledged limitations, Aerospace found an increasing trend away from applying flight-like testing methodologies where they were previously considered routine, such as in unit and subsystem performance testing and software compatibility with hardware in the loop. Similarly, intersegment testing between the ground and space segments was often eliminated entirely or greatly reduced in scope.

The decision to omit or scale back these tests must be accompanied by a clear assessment of the attendant risks. Where there is significant risk exposure by not being able to test appropriately, mitigation strategies can be developed early in the life cycle. Aerospace found these risks were not always well understood, and consequently, mitigation strategies were not effectively applied.

Aerospace found that best practices for flight-like testing had not been codified in the industry. There was a general lack of practical guidance for determining how well or poorly the testing was conducted. This was particularly true for "day in the life" operational testing.

Traditionally, issues and problems uncovered during satellite development and testing would result in design and process changes, which would in turn be scrutinized for insights that very often improve the development and verification process. As a third-party observer, Aerospace could look across contractor boundaries and identify key lessons and practices, which could then be used to help prioritize the reintroduction of industry-wide specifications and standards. With the cancellation of these standards in the mid-1990s, contractors were left on their own to accommodate technological changes and lessons learned into their own processes—with variable success.

A Leap in Complexity

While verification rigor had dropped, overall satellite complexity rose, often exponentially, as a result of advances in electronics technology and software. Not only were these systems using more parts, but the parts themselves were often far more complex, requiring much more stringent design verification and qualification practices. The greater use of field-programmable gate arrays (FPGAs) and application-specific integrated circuits (ASICs), with millions of embedded transistors on a single device, poses an even greater testing challenge.

Not only does increasing complexity pose a challenge to the verification process, but it also implies an increase in the likelihood of latent design and workmanship defects. Given the increases in complexity, the corresponding pressures on the verification processes, and the increased failure potential, the industry and government had embarked on a path of conflicted logic that resulted in numerous problems that were often not detected until late in development cycle, or even on orbit.

Under acquisition reform, the government did not always specify requirements for qualifying the parts used in space systems. The manufacturers assumed responsibility for piece-part qualification, based on the application and the performance requirements at the system level. This led to problems for several reasons.

This chart shows the value in dollars of U.S. space assets lost during the 1990s. Recent independent studies have shown that reducing technical verification rigor and diminishing the role of independent technical oversight in the development of government and commercial space systems results in greater problems, as evidenced by higher failure rates and cost and schedule overruns (view larger image).

Acquiring qualified parts had become more difficult as suppliers focused on commercial markets at the expense of the military space market (which, although relatively small, typically requires stricter parameter control, higher reliability, wider temperature ranges, higher dynamic response, radiation hardness, and similar traits). In addition, as suppliers switched from a product qualification model to a process qualification a model, both primary contractors and government lost insight and traceability into parts because suppliers were not required to provide technical data for qualification and traceability. The government had even less insight, with fewer people to track problems and less oversight into manufacturing details.

Cost and schedule assumed a greater role in determining which tests and analyses should be used to demonstrate that a device was acceptable and could meet system requirements. Because of inadequate resources and shifting priorities, only new or problematic suppliers were evaluated or closely monitored. Verification of compliance was less disciplined for subtier contractors, and the prime contractor's role changed from "right of approval" to "right of rejection."

Flight software complexity had increased even more, and it is now statistically impossible to find all possible defects in large software systems. Despite continuing advances, debugging code remains time-consuming: up to 50 percent of a programmer's time can be spent debugging code. Furthermore, testing requires a test plan, detailed test procedures, and scripts for providing input to an automated testing tool—an effort that can be just as prone to error as the code it purports to test. Altogether, complex software entails meticulous verification planning and software development, a challenge that is not addressed in development and budget allocations. This underscores the need for a rigorous independent assessment of interrelated software and hardware requirements development early in the process.

Today's satellite systems involve multiple user nodes. The increasing number and complexity of interfaces led to a rise in interface problems during system-level and end-to-end testing among ground, user, and space segments. These complex interfaces present a challenge to simulation tools and limit the accuracy of design-margin predictions and verification by use of models and simulations (see sidebar, Reemerging Part Specifications).

A Breakdown in Systems Engineering

In addition to finding problems with verification and testing, the Aerospace study identified numerous problems with systems engineering practices, including source selection, requirements definition and flowdown, system design, engineering requirement verification, manufacturing and integration support, and scheduling.

Data analyzed pointed to a number of systems engineering deficiencies that resulted in numerous late-build-cycle problems, highlighted by the large increase in design flaws (detected in system-level testing) since 1995. Specific deficiencies include marginalizing the peer design review process and related documentation, descoping preliminary and critical design processes, and marginalizing the risk management process. In general, Aerospace found that systems engineering processes were fragmented.

Several additional systems engineering challenges were also discovered—most notably, personnel shortfalls, flawed assumptions regarding the insertion of commercial products in a given design, less emphasis on achieving flight-like testing, and greater emphasis on cost and schedule versus performance and reliability.

Spacecraft are extremely complex, and program managers have always felt pressure to reduce costs and head count. Coupled with the aging demographics of the space industry workforce, the pressure to minimize staffing levels had decimated government and contractor systems engineering teams—sometimes depleting teams from five or six deep to one individual who may not have enough technical breadth to understand the potential impact of design issues and the many problems that occur during production. This increased the chances that design errors would go unidentified (and uncorrected) until they caused a failure. The lack of personnel also led to a reduction in oversight of the prime contractors by the government and of the subcontractors by the prime contractors. This increased the likelihood that problems caused by streamlined design and verification process changes at one level would not be communicated to another.

Another common shortfall in systems engineering and verification planning involved overly optimistic assumptions about the use of commercial off-the-shelf (COTS) or heritage components. In many cases, the developer assumed that a commercial or heritage product was suitable for a new application without giving sufficient scrutiny to the intended design use conditions. In reality, commercial or heritage products almost always require more modifications than expected, and this adversely affects program schedule. Sometimes, problems with these products were overlooked until they caused costly failures in ground testing or even on orbit because assumptions regarding the suitability of the original design to the new application's actual design environment and operational scenarios did not pan out.

A Get-Well Road Map

The Aerospace study concluded with a series of specific recommendations for the national security space community. In particular, acquisition managers must:

• strictly adhere to proven conservative development practices embodied in best-of-class specifications and standards;
• apply rigorous systems engineering, including disciplined peer design reviews and clearly traceable verification processes;
• emphasize requirements verification and testing of all hardware and software, focusing on the early development phase and lower-level unit design;
• apply updated and consistent software development and verification processes, including meaningful metrics;
• instill effective closed-loop design and communication processes, with special attention to new technology insertion, application of COTS components, and detailed assessment of operational data and lessons learned;
• strengthen the qualification and verification of parts, materials, and processes;
• develop a pyramidal and flight-like requirements verification policy and assess the risk of deviations from this policy;
• develop a set of engineering handbooks written from the perspective of the system program office;
• manage the product life-cycle data within the system program office and across the enterprise and learn from it.

When these practices are applied together throughout development, they have historically resulted in successful program acquisitions and mission success. Recommendations from major government review panels are largely consistent with Aerospace conclusions regarding the proper application of industry best practices and lessons learned. Moreover, the Aerospace study provides detailed evidence as to why national security, long-life, space acquisition—and more pointedly, the verification process—requires a different approach than that of a purely commercial space program. As a result, acquisition leaders are once again emphasizing a more traditional, proven, and disciplined approach to engineering space systems (see sidebar, A Return to Standards).

One critical part of such an approach is to ensure that appropriate specifications and standards are applied on a given contract. Specifications and standards arise from an often painful and costly evolutionary process, and in a sense, they form the embodiment of decades of lessons learned and best practices. These specifications, standards, and guidelines therefore form the cornerstone of traditional best practices that help ensure successful execution of a satellite program. Realizing this, Aerospace has already helped introduce revised and new national security space standards for space systems development, which draw upon the previously canceled military standard with enhancements to bring them up to date with current best practices (see table, National Security Space Specifications and Standards).

Additional best practices related to a successful qualification and acceptance test strategy will be defined in a comprehensive test and evaluation handbook under development at Aerospace (see sidebar, The Testing Handbook). In addition, Aerospace is developing and publishing handbooks that provide technical rationale, methodology, and tailoring guidance for mission assurance and space vehicle systems engineering.

Summary

The findings of the Aerospace study are helping spur national security space initiatives to establish more disciplined systems engineering, verification, and mission-assurance strategies. The assessment of development practice changes, together with an analysis of on-orbit and factory test failures, provided a greater degree of insight into the effectiveness of the integration and testing processes, the critical role of the systems engineering process, and the sensitivity of design and verification processes to the consequence of acquisition policy change. The study also shed new light on the relationships among test parameters, levels of assembly tested, test effectiveness, test-related fatigue, and the resulting influence on cost, schedule, and mission success.

Successful space systems in the past adhered to a rigorous requirements flowdown process that was tied to a comprehensive and disciplined verification process that ensured each requirement was properly verified and traceable to a specific test, analysis, or inspection document. By reemphasizing verification and testing at the lowest level and testing under flight-like conditions, the government is underscoring the importance of applying technical rigor in areas where conflicting and often marginally successful verification methods were being applied because of the lack of paradigmatic specifications and standards. Systems engineering and mission assurance revitalization initiatives are well attuned to the urgency to correct the lapses in the acquisition strategy and have consolidated efforts to accelerate development of a common and technically relevant set of specifications, standards, and best practices for all national security space programs.

To Fall 2005 Table of Contents