The Role of Independent Assessments for Mission Readiness
Tom Freitag and Bernardo Higuera
Independent teams—once viewed as costly and time-consuming—play a vital role in helping to ensure mission success.
In the 1980s, Air Force regulations prescribed independent reviews for all boosters and many satellite missions. Independent review teams composed of Aerospace, government, and contractor personnel would review each mission, present a risk assessment, and then disband. The teams were large and required a significant amount of contractor support. They had no permanent members, and the results of their reviews were often shared late in the launch flow process. If significant issues arose, options were limited and expensive.
In the mid-1990s, the Air Force Space and Missile Systems Center (SMC) largely eliminated these independent reviews as part of the cost reductions sought through acquisition reform. In contrast, the National Reconnaissance Office (NRO) continued its independent reviews, and in 1997 established a Mission Assurance Team composed of Aerospace, government, and contractor personnel. An early assignment for this team was to evaluate the advanced avionics and new solid rocket motor upgrade of the Titan IVB-24. At the same time, launch schedule slips to some Titan IVA missions—which had been reviewed by independent teams that had since disbanded—necessitated a supplemental review of those vehicles. The newly evolving Mission Assurance Team stepped in to review them, addressing component aging, storage, and site processing.
In late 1999, after the Department of Defense suffered several launch failures involving Titan IV vehicles, SMC and NRO formed an agreement establishing the Mission Assurance Team as the independent reviewer of all remaining Titan IVB missions, including those using the Centaur and the Inertial Upper Stages. The team's knowledge of the Titan IVB had developed to the point where it was seen as the expert on that launch vehicle.
The agreement between SMC and NRO was later extended to include the Atlas II and III medium launch vehicles. Meanwhile, SMC was developing its own process for independent review of heritage launch vehicles such as the Delta II and newer vehicles such as the Delta IV and Atlas V. These newer vehicles in particular needed multiple independent review teams because they were being developed by different contractors and designed with multiple configurations. In 2001, SMC formally resurrected its independent reviews, establishing a permanent Independent Readiness Review Team with more direct reporting lines to SMC leadership and an annual budget.
Subsequently, a decision was made to have the SMC review team focus on Atlas and Delta missions for the Air Force, while the Mission Assurance Team would focus on Atlas and Delta missions for the NRO. Both teams would share information on a regular basis.
In establishing these formal review teams, the NRO and SMC recognized the need for a dedicated core group of people who would maintain continuity throughout launch campaigns. This continuity would allow the teams to delve into specific problems requiring a high degree of investigation, analysis, and data review. Permanent membership would also reduce the number of orientation briefings that had been necessary each time a new team was formed. The government started budget planning to ensure the teams would be funded for several years, and this, in turn, stimulated long-term commitments from launch vehicle experts. It also fostered better coordination and integration with the launch vehicle contractors, which led to early and timely resolution of issues and the reduction of costly schedule slips(see sidebar, A Ten-Step Approach to Independent Review).
The Mission Assurance Team
Today, the Mission Assurance Team is an integral part of the mission certification process for the NRO. Its overall objective is to complement the more extensive launch verification performed by Aerospace. The focus is on deviations to the launch vehicle baseline, including documented nonconformances, out-of-family items, work performed out of position, and changes implemented by the launch vehicle contractor without supporting qualification or adequate verification. These include in-line design, vendor, and production optimization changes known as Class II changes. The team also identifies and evaluates hardware and mission design first-time flight items and pursues the intersection of companion hardware failures (similar hardware not assigned to the specific vehicle being reviewed) to ensure there are no latent defects present. Postflight data review ("test like you fly" screening) and acceptance testing evaluation is also an integral part of the assessments.
Aerospace provides technical leadership to the group, and the NRO provides funding and management. The Mission Assurance Team performs an in-line independent technical risk assessment of launch vehicle issues, configuration, hardware disposition, and all other areas of launch vehicle build and process that can potentially increase mission risk. This risk assessment is made on a purely technical basis, according to published guidelines, and is independent of programmatic constraints such as cost and schedule. It is a comparative analysis based on a defined and accepted launch vehicle baseline and its associated baseline reliability.
The Mission Assurance Team covers three main areas: structural and mechanical engineering, propulsion, and avionics and mission design. Although the focus is primarily on hardware, overlap into mission design, analysis, and integration helps the team evaluate the suitability of each system for a particular mission. Each main review area has 10–12 members with two leads. Half of the members serve full time, and half provide part-time support, including members of Aerospace, contractors, and expert consultants.
In the past, the NRO's Mission Assurance Team supported SMC in its independent risk assessment for select Titan, Atlas II, and Delta II launch vehicles. In fact, with the exception of the NASA Cassini mission, the team has assessed all Titan IVB vehicles for SMC and NRO, including some SMC unique Inertial Upper Stage missions. The team assessed the Atlas II MLV 7 and MLV 9–15 missions, the Delta GPS IIR-2, Geolite, and NROL-21 missions, and the Orbital Sciences Taurus/Stex mission. The technical risk assessments have evolved along with space launch vehicles. The team has moved from assessing mature Titan, Atlas II/III, and Delta II programs to looking at the new Atlas V and Delta IV rockets. Work for these new boosters, which lack an extensive historical baseline, relies on lessons learned from test failures and near misses as well as the overall incorporation of best practices shared by team members.
The Mission Assurance Team tracks challenging launch fleet issues and investigates associated risks that have been identified by Aerospace, the Launch Directorate, SMC's Independent Readiness Review Team, and contractors. For example, the teams are working together to analyze the constantly evolving defect criteria for delaminations in a graphite epoxy motor and the implementation of manufacturer's corrective actions in the production of launch vehicle batteries. The Mission Assurance Team participates in the Aerospace President's Review, the SMC Flight Readiness Review, and the NRO's Mission Certification Review, as well as quarterly program reviews to senior NRO management.
Team members work daily alongside contractors, and this interaction has created a strong understanding of the need for timely risk assessments, early involvement, and ample time to review and manage risk issues. The team's mission assurance procedures are continuously reviewed, compared with those of other independent teams, examined with respect to government studies, and redesigned as necessary.
Examples of Findings
The Mission Assurance Team supported the development and use of a structural dynamics high-frequency data-reduction and harmonics analysis tool used to screen out-of-family harmonics behavior in an upper stage engine system. The harmonics were the suspected source of some RL-10 engine failures. This analysis tool has undergone significant improvements and has since identified engines that have unusually high harmonic content in the Atlas II AC-109 and Delta IV NROL-26. The team's findings on the harmonics were presented to the appropriate contractors, who subsequently removed the engines.
Another review identified high porosity in a composite structure. This high-porosity area had escaped the original inspection, and the Mission Assurance Team presented its findings and recommended repairs. The team also identified an improperly qualified acceptable defect size for composite structures, and recommended additional testing to validate the allowable defect sizes. The contractor performed additional testing and concluded that the drawings should be modified in the majority of the structures to reflect a new, smaller allowable defect, which was one-quarter the size of the original defect.
A review of hardware photos during vehicle assembly identified potential scuffing or chafing of a wire harness on a payload adapter. Because this phenomenon was implicated in a prior booster failure, the Mission Assurance Team promptly brought the findings to the contractor's attention. The contractor added standoffs and additional wire-harness protection, and implemented new criteria to ensure proper standoff between wire harnesses and structures.
A standard review of a solid rocket motor included an allocation for the twisting of a composite motor case resulting from the internal motor pressure, which is defined as case twist. Mission Assurance Team engineers identified the proposed case stack as an out-of-family worst-case twist, which exceeded the twist on all previously flown missions. The case twist had the potential for generating additional internal loads that were not accounted for in the stress analysis and load calculations for the vehicle. The contractor accepted the team's recommendation and rearranged the solid rocket motor stack to minimize case twist.
The Independent Readiness Review Team
The Independent Readiness Review Team performs risk assessments of space launches and reports findings in prelaunch reviews to the SMC commander. Aerospace provides technical leadership for the group, supplying 10 full-time employees on the team; these are augmented as necessary with additional Aerospace engineers and industry contractors. The Independent Readiness Review Team is a matrix-style organization with system and panel leaders. Each system leader is responsible for the review of a specific system. Each panel leader supports multiple system leaders in specific disciplines such as propulsion, avionics, software, and mechanical and structural engineering.
The Independent Readiness Review Team uses this chart to assess risk, probability, and potential impact to a mission as issues are raised during reviews (view larger image). |
Primary objectives include identifying technical risks, making recommendations for mitigation, and providing independent assessments of launch readiness. The team participates in space program development, including technical interchange meetings, integrated product team meetings, hardware acceptance, and pedigree and design reviews; it also examines selected parts, components, subsystems, and compliance documentation. The reviews usually start on a satellite two years before a launch and on boosters one year before launch. The team has performed independent reviews of the Atlas V and Delta IV boosters as well as heritage boosters such as the Titan II, Delta II, Pegasus, and Minotaur. The team also reviews payloads for the Defense Support Program, Global Positioning System, Space-Based Infrared System, Defense Meteorological Satellite Program, Military Strategic and Tactical Relay System, Advanced Extremely High Frequency system, Wideband Global Satcom System, and selected Space Development and Test Wing flights.
These independent reviews support the SMC flightworthiness certification process and are based on published guidelines such as SMCI 63-1201, Assurance of Operational Safety, Suitability & Effectiveness (OSS&E) for Space and Missile Systems; SMCI 63-1202, Space Flight Worthiness; SMCI 63-1203, Independent Readiness Reviews; and SMCI 63-1204, Readiness Review Process. The Independent Readiness Review Team also evaluates each mission-specific space program office and the contractor processes, and, as necessary, offers opinions and recommendations for improvement. Team members participate in problem resolution and failure investigation activities, such as nozzle delamination investigations, solid rocket motor redesign, and satellite mechanism failure resolution activities. The team formally presents its results at periodic program reviews with SMC leadership, at the Aerospace President's Review, and at the SMC Commander's Flight Readiness Review.
Examples of Findings
The Independent Readiness Review Team's software group identified severe shortcomings in flight software development and execution for an SMC program. The team created a risk-reduction road map that the system program office adopted and has been using ever since to track risk reduction. The team also uncovered grossly inadequate unit-level tests for flight software and recommended remedies that have since been implemented by the program office and contractor.
In another investigation, the team diagnosed the root cause of a failure in a phase-locked loop circuit in a microwave imager sounder and recommended corrective action that was implemented by the program office and contractor. By examining schematics, a review engineer identified the location of an open circuit that was causing intermittent operation of the phase-locked oscillator. The engineer visited the contractor's facility and examined the hardware firsthand. The engineer was able to reproduce the anomalous behavior by pressing on the transistor lead identified during the schematic review. While diagnosing this problem, the review team also discovered severe workmanship problems. These were corrected, and the payload launched. It continues to perform as intended.
The review team's structural and mechanical engineering panel provided recommendations to reduce the risk of latch failure in a telescope's protective contamination cover, which represented a single point of failure. The program office and contractor accepted the design and test recommendations to reduce the risk of mission loss.
During booster component reviews, the team identified two motors with suspect nozzles. The first had a nozzle that was dropped off a pallet and later installed. The review team found that the analysis to justify the use of the dropped nozzle was inadequate because it did not consider dynamic loads. The motor manufacturer and the program office agreed that additional work was required to clear the nozzle for flight. The second motor had two separate sources of polyacrylonitrile tape used to manufacture the carbon-phenolic exit-cone liner. Tape from one source was used on half of the liner, and tape from another source was used for the remainder. The two tapes had mechanical properties that differed by as much as a factor of 2. The motor manufacturer had not done any analysis to determine the effects of using an exit-cone liner with mechanical properties that changed dramatically midway through. The only prudent course was to replace the motor and set it aside until an adequate structural analysis of the exit-cone liner could be performed.
Another issue involved the lack of traceability to a qualified baseline for a solid rocket motor. The team reviewed 11 ship sets, including several that had flown, to create a baseline and evaluate flight risk. Results showed that each change, while not necessarily qualified by SMC standards, was reasonably supported by flight test analysis or experience from other programs. The launch proceeded with an elevated risk rating and was successful. The Independent Readiness Review Team has since recommended that the contractor show proper traceability to the qualification baseline, or perform additional qualification tests as needed.
Conclusion
In 2006, Boeing and Lockheed Martin established United Launch Alliance (ULA), a joint venture to produce the Atlas V, Delta II, and Delta IV launch vehicles. This has allowed the Mission Assurance Team and the Independent Readiness Review Team to blend their Atlas and Delta review functions, which had to be isolated when the two contractors were competitors. These unified teams have even stronger processes for communicating lessons learned across programs, and the ability of team members to discuss and transfer experiences has proved highly effective. However, the merger presents a new challenge: Much of the engineering and manufacturing for the Delta and Atlas will be moving from Huntington Beach, California, to ULA headquarters in Denver, Colorado, for engineering, and to Decatur, Alabama, where the launch vehicles are currently built and assembled. The logistics of these new arrangements make the need for independent review all the more critical.
The Independent Readiness Review Team and Mission Assurance Team have become accepted members of the mission assurance process. Although some of the specific processes differ, the objectives and products of each team are highly valued by their sponsors. As the failures of the 1990s recede further into the past and budgets receive greater scrutiny, there is a danger that mission success may lose its emphasis. The results would be disastrous. Since 2001, the NRO and SMC have experienced an unprecedented string of successful launches, and this trend has been sustained through an unyielding commitment to mission success. The continued focus on operational safety, suitability, and effectiveness—including the use of independent teams to minimize risk—must not be marginalized or eliminated.
To Fall 2007 Table of Contents
