A Mission Assurance Toolbox
Jim Roberts, Bruce Simpson, and Sergio Guarro
Aerospace has developed a suite of specialized tools that facilitate technical analysis and program assessment in support of mission success.
As anyone who has undertaken a home remodeling project knows, the proper tools can make the difference between finishing in a timely manner, with a product that will be a source of pride, or having to make repeated trips to the store for additional tools and materials.
Aerospace is entrusted to provide objective assessments of national security space programs and recommendations for addressing potential problems. Through the years, Aerospace has developed tools to support this effort. Some have proved useful for specific problems, while others could be adapted to a wide range of issues and have survived the test of time and repeated use. The more functional tools have been continually improved and adapted to serve more people and a greater variety of tasks.
To support the broader goal of mission assurance, these tools must be flexible, easy to use, adaptable to a diverse set of customer requirements, and capable of being tailored to the needs of a specific program. They must also have the capacity to store, organize, and present data in a format that easily facilitates comprehensive and reliable analysis and evaluation.
The following is by no means a comprehensive list of tools available to Aerospace personnel, but represents some of the more frequently used tools.
The functional flow diagram outlines the primary elements of the Aerospace independent launch-readiness verification process, as embodied in the Launch Verification Matrix. This comprehensive process extends from concept and requirements definition through flight operations and includes a postflight assessment. |
Launch Verification Matrix
The Aerospace launch verification process independently determines launch system flight readiness. It is a capability unique to Aerospace that has been employed for more than 40 years. This process extends from concept and requirements definition through flight operations; it entails the detailed scrutiny of hundreds, if not thousands, of components, procedures, and test reports; it draws upon independently derived system and subsystem models to objectively validate contractor data; it provides timely review through firsthand involvement in all aspects of the launch campaign; and it concludes with a thorough postflight assessment using independent analytical tools and independently acquired telemetry data to generate useful feedback and monitor performance trends.
The activities encompassed by this process are defined by approximately 2000 specific tasks. Aerospace codified this task list in 2001, identifying completion criteria, categorizing the rigor with which a task would be executed (from basic monitoring through full independent analysis), and establishing execution priority (based on mission criticality).
These tasks were initially presented in a spreadsheet, referred to as the Launch Verification Matrix. As the value of this matrix was recognized and the number of missions grew, more engineers needed access to record their launch verification information. So, the matrix moved to a database, which was subsequently made available to approved users via a Web server maintained by the SMC Launch and Range Systems Wing.
The database contains the verification tasks, their completion criteria, the rigor with which the task is executed, the completion date, the responsible engineer's evidence of completion, and other tracking information. Responsibility for each task is assigned to two people, an Air Force engineer and an Aerospace or SETA engineer, who work together throughout the mission to complete the task, recording status information, issues, and completion rationale.
A mission's launch verification tasks are selected from the general task list based on the launch vehicle configuration, mission requirements, and first-flight items. The task set is reviewed by the responsible engineers, their management, and the vehicle chief systems engineer for appropriateness to the mission. Upon concurrence by the Launch and Range Systems Wing program manager and chief engineer, the cognizant Aerospace principal director, and (for NRO missions) the Office of Space Launch mission manager, the matrix is presented to the Launch and Range Systems Wing Configuration Control Board for approval. Frequently, changes during the launch campaign dictate changes to the launch verification plan. When changes that could affect the program occur (such as changes to the Aerospace staff budget, launch schedule, or launch verification tasks), they are made to the Launch Verification Matrix with approval by the Configuration Control Board. This mission-specific database is maintained throughout the postflight review and final mission report.
The user interface for the Launch Verification Matrix provides several features in addition to displaying task status to the responsible engineers. Among these are user reports, facilities to track launch readiness issues to be resolved before flight, hardware pedigree, parts/serial number database, and several management reports and metrics as well as the necessary infrastructure facilities to set up and manage the mission databases. Management reports of note include task completion status, delinquencies, upcoming task deadlines, and graphical representation of key management information. Most reports can be exported to a spreadsheet for further manipulation. Additionally, databases from different missions are linked together for easy viewing of history on a particular launch verification task.
The spectrum of launch verification activities is accomplished by a cadre of engineers with expertise in a wide variety of disciplines, including systems engineering, mission integration, structures and mechanics, structural dynamics, guidance and control, power and electrical systems, avionics, telemetry, safety, flight mechanics, environmental testing, computers, software, product assurance, propulsion, fluid mechanics, aerodynamics, thermal engineering, ground systems, and facilities and operations. These engineers document in the Launch Verification Matrix their efforts and issues through all phases of launch vehicle development and operations. This information provides the basis for the launch readiness verification letter that Aerospace delivers to the Air Force before launch.
The Launch Verification Matrix has become the cornerstone of launch vehicle mission assurance. It establishes the launch readiness verification plan for a mission; it provides a repository for the launch readiness verification information during the launch campaign; and it captures historical documentation for future anomaly resolution. It identifies that set of mission assurance activities that cannot be compromised and that must not be traded against cost, schedule, and performance. It has been a key tool for Air Force and Aerospace launch verification management on multiple EELV missions. It is currently in use to manage upcoming Delta IV and Atlas V missions and to support legacy Delta II missions.
The Mission Assurance Guide
The Mission Assurance Guide was developed to provide basic mission assurance documentation and guidance and to reinforce the corporate system engineering and mission assurance functions and accountabilities, aligning them with the SMC/NRO systems engineering revitalization efforts. The guide defines a baseline of six core mission assurance processes and seven support disciplines in terms of executable functions. It identifies the major objectives, tasks, and techniques associated with these processes and disciplines, referring to Aerospace systems engineering and test engineering handbooks as well as to current standards for technical task guidance. Core processes are: requirement analysis and validation; design assurance; manufacturing assurance; integration, test and evaluation; operations readiness assurance; and mission assurance reviews and audits. The support disciplines are: risk management; reliability engineering; configuration management; parts, materials, and processes management; quality assurance; systems safety assurance; and software assurance.
The guide is structured to allow the user to develop a plan for applying these processes and disciplines in a flow of executable tasks applied against specific work breakdown structure (WBS) elements. The Mission Assurance Guide is designed to be used in conjunction with the Mission Assurance Verification Matrix, which provides users with additional task details and implements that support the task tailoring, execution, and assessment needs of specific programs. (For more about the guide, see the companion article in this issue.)
Mission Assurance Verification Matrix
The Mission Assurance Verification Matrix (MAVM) is a database and software tool that defines, documents, and supports the implementation of the guidance contained in the Mission Assurance Guide. It provides a full definition of the mission assurance core process and support-discipline task structures and execution flows, organized by program acquisition phase, hierarchical relationship, and WBS element. It also defines and documents each individual mission assurance task, providing specific information that supports associated planning and execution activities—including the assessment of risk associated with planned and actual levels of task criticality and depth of execution.
The MAVM, like an early version of the Launch Verification Matrix (upon which it was modeled), is based on a Microsoft Access database platform. It uses the relational database features of Access to record and document more than 2000 mission assurance tasks according to acquisition-phase sequence of execution, core-process and support-discipline structural hierarchy, and association with specific WBS elements. Its task documentation, editing, and assessment features enable the execution of the entire cycle of mission assurance planning and execution that is defined and prescribed by the Mission Assurance Guide. More specifically, it facilitates the execution of the following key implementation steps by users supporting mission assurance functions in a specific space program:
Mission assurance plan definition and tailoring. This implementation step is supported by the MAVM capability to edit (add, delete, copy, and move) mission assurance tasks within the process and discipline structures defined in the guide. In this fashion, task groups can be tailored to meet specific program needs and constraints and can be associated with specific elements of the program WBS. At completion, this tailoring activity results in the definition of a program-specific mission assurance plan.
Mission assurance plan risk assessment. This step produces an evaluation of the overall adequacy of the mission assurance plan in the form of a "plan risk assessment." To carry out the assessment, the user rates each task associated with a WBS element in terms of its "criticality" (i.e., importance, relative to other tasks) and intended depth of execution (i.e., the amount of time and personnel assigned to it), which in turn generates a "potential risk exposure" rating for the task. A risk roll-up algorithm that proceeds upwards in the hierarchical task and WBS structures also produces risk ratings for groups of tasks and entire WBS areas. The potential risk exposure rating permits a rapid engineering evaluation of the adequacy of resources assigned to validation and verification in all significant program areas. It also enables, in relative terms, a judgment of the overall balance of the mission assurance plan. If inadequacies are identified in either sense, the plan-tailoring step can be revisited to achieve a better risk exposure balance.
Detailed mission assurance task definition, documentation, and tracking. Each task record in the MAVM contains fields for storing detailed information concerning the task. This includes, but is not limited to, definitions, active links to reference and guidance documents, identification of related tasks and checklists, task closure/completion criteria, criticality and risk ratings, accountable personnel, and planned and actual schedule. The sum of this information permits the full definition and tracking of the activities associated with the execution of each individual task.
Mission assurance task execution assessment. The MAVM records task completion information and the degree to which task closure criteria have been satisfied. Together with the information on task criticality, which can be further updated to reflect the most recent program developments, this is used to produce an assessment of the residual risk after task execution. As with the earlier potential risk rating, the residual risk rating for each task can be rolled-up to produce residual risk ratings for groups of program WBS elements and entire program areas and can be used to support decisions concerning program and mission readiness and launch certification, including the question of whether additional assurance activities are warranted in any particular area.
A new implementation of the Mission Assurance Verification Matrix, called "iMAT," is being developed. It will employ a Web-based tool and a more robust Oracle database. Early versions of this tool are being piloted in an Aerospace program office, with plans to introduce the tool to two or more additional program offices in 2008. The iMAT software currently in use is in alpha-stage testing, with beta testing scheduled to begin in early 2008. The goal is to produce a flexible and efficient tool—deployable and controllable at the corporate level—that Aerospace personnel can use to effectively implement the principles and tenets of the Mission Assurance Guide, providing program managers with information and assessments that assist their decisions concerning any level of mission assurance in any phase of the acquisition process.
The Mission Assurance Verification Matrix defines and documents the tasks set forth in the Mission Assurance Guide, providing specific information that supports associated planning and execution activities—including the assessment of risk associated with planned and actual levels of task criticality and depth of execution. |
The Mission Assurance Portal
The Mission Assurance Framework, Guide, and Verification Matrix all reside on an internal Web site known as the Mission Assurance Portal. The site provides ready access to a diverse set of information and data to support mission assurance efforts. The tools enable Aerospace engineers and scientists to answer questions pertaining to specific missions.
For example, the site contains the latest approved versions of NRO and Air Force acquisition policies and supporting documentation. The site also provides access to the approved lists of specifications and standards for SMC, NRO, and MDA as well as a list of specs and standards compiled by the Aerospace Specs and Standards Community of Practice. A "lessons learned" section provides access to numerous Aerospace reviews covering a wide range of topics. Similarly, the "best practices" area lists Aerospace technical reports, memoranda, and briefings on topics ranging from acquisition strategies to hardware and software testing. There is also documentation on addressing items in the Data Item Descriptions—the government-issued documents that define the data required of a contractor. A resource directory contains a list of functional area experts and a list of corporate databases. In addition, the site contains a document search engine, a catalog of Aerospace reports, and a repository of 45 technical handbooks primarily produced by Aerospace, the DOD, and NASA.
One of the goals of the Mission Assurance Portal is to facilitate the distribution of corporate mission assurance information to the relevant Aerospace user community. A great example of this is launch history data. There are numerous users of launch history data throughout the company with varying objectives and needs. Although Aerospace maintains a validated launch history data set in the Space Systems Engineering Database, many users maintain their own independent data sets or use public sources.
The portal developers interviewed various stakeholders and determined the features and requirements that would assist them in completing their analysis on launch history. These interviews revealed a strong need for a tool that would provide graphical charting and trending capabilities for high-level analysis. Also, users wanted to run ad hoc queries and searches, without predefined views, and to download the results to a spreadsheet for further manipulation. Based on this feedback, the portal developers created a new tool that will be tied in to the corporate Space Systems Engineering Database and coordinated with the Space Launch Office. It allows users to quickly specify parameters of interest, generate a variety of charts and graphs, and download the results.
The Aerospace Problem Notification Processes
In the early 1970s, Aerospace developed an internal problem notification process because information pertaining to nonconformance in parts, materials, and processes was not being effectively disseminated to all the programs that might potentially be affected. In addition, Aerospace realized that the Government-Industry Data Exchange Program (GIDEP) for problem notification needed supplementing, primarily because it suffered from long delays and could miss problems with suppliers that were not GIDEP members.
The Aerospace process relies on three types of documents to disseminate information pertaining to nonconformance of parts, materials, and processes: problem advisories, product/process alerts, and experience-sharing bulletins. Problem advisories provide early notification of nonconformance and contain preliminary information such as a description of the nonconformance, the supplier identity, part number, and lot date code or other unique identifier to allow programs to assess the impact on their hardware. Product and process alerts formally document an issue and contain additional information not found in problem advisories, such as the root cause of the nonconformance, recommendations to correct it, and recommendations to prevent its recurrence. Experience-sharing bulletins are used to document and share lessons learned from the resolution of nonconformance issues.
To modernize the process and to support the SMC/NRO alert systems, Aerospace created an internal Web site that allows any employee to create an alert bulletin. The document is automatically routed through the approval cycle. Once it is approved, a link is sent to all programs via e-mail. The site offers search capabilities for unique identifying information, summarization of alert bulletins for a time period, and discussion threads for interaction between the originator, the Parts, Materials, and Processes Department, and the affected programs. A copy of the bulletin is stored in the corporation's electronic archives.
Other Tools
In addition to the tools presented here, Aerospace has created numerous others for a variety of programs. For example, the Spacelift Telemetry Acquisition and Reporting System (STARS) is used to record real-time launch-vehicle telemetry for postlaunch analysis; it's been instrumental in achieving the current high rate of successful defense program launches. The Continuous Aerospace Risk Management and Assessment (CARMA) tool enables consistent implementation of risk management and assessment processes across a broad spectrum of programs; it is made up of a database and associated analytical modules that can be applied at the program level or within the context of specific mission assurance activities. There's also the Environmental Test Thoroughness Assessment tool, the Reliability Network Analysis tool, and the Satellite Orbital Analysis Program (SOAP). These and other tools are accessible via the corporation's internal Web site, and most will ultimately be available via the Mission Assurance Portal.
Conclusion
The ultimate goal is to enable all employees to search for the data and information they need to support their customers. The Mission Assurance Portal, in the future, will provide links to these and other tools identified by the corporation as beneficial to the goal of mission assurance. In addition, the portal will provide links to corporate subject matter experts—people recognized by the corporation and their peers for their professional expertise in their field. Aerospace is committed to reducing the time and effort spent searching for information so that staff members can devote more of their efforts to analysis and problem resolution.
As it is at most corporations, the workforce at Aerospace is subject to regular and continual change. Employees retire or leave, and new employees replace them. Some come with experience, others arrive fresh out of school, eager to learn and make an impact. The goal of the corporate toolset is to provide all these scientists and engineers with a standardized methodology for analyzing programs, answering technical questions, suggesting improvements, and resolving difficult problems.
Acknowledgments
The authors thank Art McClellan and Mark Goodman for their contributions to this article.
To Fall 2007 Table of Contents
