The Use of Commercial Software in Ground Systems Development

Richard J. Adams and Suellen Eslinger

Recent acquisition policy directives assumed that the use of commercial software would reduce costs and shorten development time. When the expected benefits failed to materialize, Aerospace stepped in to find out why.

Beginning in the early 1990s, the policies and regulations that govern the acquisition of launch and space systems began to change. The government and the Department of Defense (DOD) instituted a series of new policies and directives that they believed would improve the acquisition process and lead to less expensive but more capable systems that could be developed more quickly. One such policy change addressed the use of commercial items in new defense systems. The policy, outlined in a letter from the Office of the Secretary of Defense dated July 14, 2000, stated, "We must expand the use of commercial items in Department of Defense systems so that we can leverage the massive technology investments of the private sector and reap the benefits of reduced cycle times, faster insertion of new technologies, lower life cycle costs, greater reliability and availability, and support from a robust industrial base."

This policy was translated into development contracts that strongly encouraged the use of commercial off-the-shelf (COTS) products in developing systems for Air Force Space and Missile Systems Center (SMC) and National Reconnaissance Office (NRO) programs. But as SMC and NRO programs began to execute these contracts, they found that achieving the expected benefits was far more difficult than anticipated. As more acquisition programs began to experience COTS-related problems, it became clear that the implications of this policy were not fully understood. To address the situation, SMC asked Aerospace to perform an in-depth study of actual COTS-based systems development and sustainment experiences. The goals were to develop an understanding of the risks inherent in the acquisition of COTS-based systems, to identify lessons that could be applied to other programs, to derive recommendations for improving acquisition performance, and to share those lessons and recommendations across SMC and NRO programs.

After an initial analysis, Aerospace determined that the problems experienced by SMC and NRO were primarily associated with ground systems, and within the ground systems, the problems were almost exclusively related to COTS software. As a result, the Aerospace study focused on understanding the problems associated with incorporating COTS software into ground systems. The process comprised four steps: gathering information, synthesizing lessons learned, developing acquisition recommendations, and disseminating the information for application on development contracts.

Gathering Information

Aerospace began by conducting focused interviews and documentation reviews. A questionnaire was developed to provide a framework for the interviews. It was sent in advance to the interviewees so they could prepare for the type of information being requested. The interview itself was then allowed to proceed as a free exchange of information, with the questionnaire being used to return the conversation to the topic, if necessary. The questionnaire was also consulted toward the end of the interview to identify areas that had not been addressed. In addition, interviewees were asked to provide any written documentation prepared by their program on COTS software experiences or lessons learned.

The questionnaire requested information on both good and bad experiences with COTS software. For each experience, respondents described the nature of the experience, where in the program life cycle it occurred, the software involved, the functions provided by the software, and the criticality of those functions to the system. For good experiences, respondents were asked to identify any actions taken that contributed to the positive outcome. For bad experiences, they were asked to identify any actions taken to resolve the problem or mitigate further risk, and provide information as to what they would have done differently, given the benefit of perfect hindsight.

Aerospace interviewed more than 50 representatives from 18 SMC and NRO programs, including government, contractor, and Aerospace personnel. In addition, Aerospace researchers with expertise in critical ground system domains (such as computer security and telemetry processing) were consulted about their experiences with COTS software.

Synthesizing Lessons Learned

From the interview notes and document reviews, Aerospace identified more than 150 distinct findings, which were then subjected to a series of analysis and synthesis iterations using the affinity clustering technique derived from the Total Quality Management and Six Sigma process-improvement methodologies (see sidebar, Affinity Clustering). Each iteration produced more general and abstract summaries of the 150 initial findings. Finally, six significant lessons emerged, and these would form the basis for acquisition recommendations applicable to multiple SMC and NRO programs.

Lesson 1: Critical aspects of COTS-based systems development and sustainment are out of the control of the customer, developer, and user. Vendors are market driven, and the commercial world, rather than the military, is their primary market. Thus, their primary market may diverge from military needs. Moreover, vendors' strategies and market position may change. They may go out of business, or merge with other companies. They may drop or deemphasize products or hardware platforms without warning. They focus on new features to attract new customers, and may change the type and quality of customer support. The quality of new product releases is unpredictable, and when bugs are found, vendors prioritize fixes for their primary commercial customers. They may change or drop promised features and eliminate backward compatibility with earlier versions. New features may degrade performance or create incompatibilities with other vendors' products, and such problems might not be found because vendors focus product testing on newly added features and perform limited regression testing. Vendors' fees and license structures may also change without warning.

National security space acquisition model. The first division is contract award preparation, the selection of the contractor team that will be responsible for the launch vehicle or space system development and, after contract, technical management of the work that results in the actual launch and space system capabilities.

Lesson 2: Full application of system and software engineering is required throughout the COTS-based systems life cycle. The systems and software engineering disciplines are still required for system development. Using COTS software only shortens the development portion of the software life cycle. Because commercial software has many customers, it is more general in purpose and, therefore, requires engineering analysis to determine if the general requirements are adequate to meet military needs. The COTS-based systems architecture must be flexible enough to support software evolution and replacement. The substitution of one COTS product for another, even from the same vendor, is never transparent. As COTS products evolve or are replaced, their computer resource needs change (for example, they may require more processing power and memory); therefore, adequate growth margin must be designed in. Prototyping must be engineered in a way that simulates how the set of COTS products will be used in the actual system to understand how they behave together. Vendors design their products to work alone, not as part of a larger system, and the various products chosen may interfere with each other and degrade system performance or cause other unintended effects. COTS-based systems must also be engineered to withstand product safety and security problems. Commercial vendors focus on the less stringent safety and security standards of the commercial marketplace. Without proper engineering, the vulnerabilities of the COTS products may result in vulnerabilities in the COTS-based system itself.

Lesson 3: COTS-based system development and sustainment requires a close, continuous, and active partnership among the customer, developer, and user. The benefits of COTS products may need to be traded against cost, schedule, performance, and operations and maintenance concepts. Serious product limitations or incompatibilities may be discovered at any time. The use of COTS products may require reengineering of operations and maintenance procedures as well as a relaxation of some military requirements.

Lesson 4: Every COTS-based system requires continuous evolution throughout development and sustainment. Maintaining currency with COTS software upgrades is essential during both development and sustainment. Because vendors only support a limited number of past releases, delaying upgrades may result in unsupported products and may exacerbate system impacts when they are finally incorporated. Some upgrades rely on previous versions, and this may require upgrading each previous version to get to the latest version. Upgrades in one product may cause incompatibilities in another. This will not be discovered quickly if COTS products are not upgraded on a regular basis. COTS products may need to be replaced or added at any time. The vendor may eliminate support for the product or replace it altogether. Similarly, the product may be found to diverge too much from system requirements, or contain an unacceptable limitation or vulnerability.

Lesson 5: Current government and contractor processes must be adapted for COTS-based systems acquisition, development, and sustainment. Existing software and systems engineering processes were developed prior to the new emphasis on COTS products. They must therefore be changed to reflect the new approach. They must incorporate robust COTS software evaluation and selection criteria as well as COTS software safety and security analysis. They must be able to handle product upgrades during development and allow designers to circumvent any identified vulnerability. They must also handle the more complex configuration management requirements imposed by COTS products. Developers need to reallocate their time and effort, allowing more time for COTS product evaluation, prototyping, and analysis and more time for system integration. The acquisition and sustainment processes also must be changed to permit the prioritization and reprioritization of system requirements over time. They must allow for contract milestones that are compatible with COTS-based systems development characteristics. They must accommodate flexible and efficient responses to cost and schedule changes caused by unexpected problems in COTS products and delays in the release of new versions.

Aerospace has incorporated the results of the COTS software study into the latest version of the best practices for software acquisition. Adhering to these best practices can improve a program's ability to achieve the benefits envisioned by acquisition policy makers.

Lesson 6: Actual cost and schedule savings with COTS-based systems development and sustainment are overstated. There is a universal tendency to underestimate the time and money needed for COTS-based system development and sustainment. Planners typically overlook the effort needed to develop an in-depth understanding of each COTS product and to install and configure them in the development and operational facilities. They underestimate the effort needed to perform integrated COTS product prototyping and to prepare integrated system training and documentation in addition to the COTS product training and documentation. They also fail to consider the work needed to incorporate COTS product upgrades—including installation, changes to noncommercial portions of the system to support the upgrade, regression testing, and operator training. Systems and software engineering, system integration and testing, and the costs of license fees and on-site support are also prone to faulty assumptions. Incompatibilities, performance problems, unexpected vulnerabilities, and vendor problems can occur at any time, so cost and schedule estimates must include sufficient margin to handle any eventuality.

Acquisition Recommendations

Based on these six core lessons, Aerospace generated a set of recommendations that would mitigate the identified risks in COTS-based system development and sustainment.

To begin with, program managers should explicitly consider COTS software when defining the acquisition strategy. In doing so, they must determine how strongly the use of COTS software will be emphasized, how the software will be maintained, and how cost and schedule uncertainties will be managed.

Program managers must adapt the acquisition processes to accommodate the effects of using COTS software. This entails establishing processes that promote a close, continuous, and active partnership with the developer, with explicit reference to requirements prioritization and reprioritization. The request for proposal should include language that supports the COTS software strategy. For example, the statement of objectives should include the goals and rationale for using COTS software. Similarly, the evaluation factors and proposal preparation instructions should include the criteria to evaluate how well the contractor's approach to COTS software meets government needs. The request for proposal should include a requirement that bidding contractors be independently evaluated for their ability to handle COTS software issues.

Lastly, program managers must incorporate checklists based on this study in the proposal evaluation process to determine if the bidders have considered all costs associated with using COTS software. The development contract should contain language that promotes the mitigation of risks associated with using COTS software. For example, the contractor should be asked to incorporate a software upgrade strategy during development and to perform analyses to develop a system that balances COTS software risks against new development. The contractor should also be required to implement an integrated product and process development approach and to apply systems and software engineering to the full development life cycle—even with the use of numerous COTS software products. The contractor should also be directed to place additional emphasis on system safety and security when COTS software is used and to supply operations and maintenance documentation at the system level, not just the COTS software level. During contract execution, program managers should actively participate in the developer's evaluation of COTS software, prototyping efforts, and analysis of unexpected problems caused by vendor actions, software deficiencies, and product incompatibilities.

The control room at the RDT&E Support Complex (RSC) at Kirtland Air Force Base, Albuquerque, New Mexico. This installation makes efficient use of COTS software. (Photo courtesy of US Air Force)

Spreading the Word

The Aerospace study on COTS software in ground systems resulted in concrete recommendations based on accurate and objective analysis. However, Aerospace recognizes that for the recommendations to be implemented, additional criteria must be met.

To begin with, programs that might be affected by COTS software problems must know about the study and its recommendations. They must understand that the recommendations were derived from valid data using methods that are generally accepted in the community. They must believe that the problems present a serious risk to their program's success, and that the recommendations will indeed reduce that risk. Furthermore, program managers must have confidence that, if they want to adopt the recommendations, they will have both the guidance and the resources they need.

Failure to understand and address the implications of the new commercial item acquisition policy resulted in serious difficulties during actual program execution. For this reason, Aerospace took special care that the study, its methods, and its recommendations met the above criteria. In particular, Aerospace ensured that the results of this study were provided to as wide an audience as possible within SMC and NRO. Aerospace also presented this information at conferences and in publications that went far beyond these communities.

Aerospace is now working with government system program offices and contractors for ongoing programs to identify methods of implementing these risk-reduction recommendations based on where each program is in its development. In addition, Aerospace is working with new program acquisitions to ensure that these recommendations will be appropriately included in the request for proposal and final development contract. Aerospace has also received requests from other organizations for additional information on how these recommendations could be applied to their acquisitions.

Conclusion

The acquisition process is highly complex and volatile. Aerospace plays a critical role in keeping current with acquisition policy changes as they occur, participating in the development of acquisition policy and direction whenever possible, developing guidance on the interpretation of acquisition policy, and recommending ways to apply the policy to individual programs. In addition, Aerospace maintains a collection of "acquisition best practices" spanning the entire acquisition life cycle for software-intensive launch and space systems. These best practices emerge from both good and bad experiences on SMC and NRO acquisitions.

The results of the COTS software study have been absorbed into the latest version of the best practices, which are being taught through The Aerospace Institute, briefed to individual programs, and presented at conferences to help the government understand the implications of changing DOD acquisition policy. Adhering to these best practices can improve a program's ability to achieve the benefits envisioned by acquisition policy makers.