Independent Assessments for NASA

Debra L. Emmons and Robert E. Bitten

The Aerospace Corporation has developed a suite of program and project assessment capabilities that provide input to NASA strategic decisions.

NASA's goal to provide a human presence in space while contributing to the knowledge of the science of Earth, other planets, the solar system, and the universe requires a diverse set of scientific and exploration missions. Its successful development of these portfolios depends upon a sustainable and affordable long-term strategy. To provide a basis for an adequate annual funding profile to fit within NASA's budget, an objective assessment of a mission's technical baseline, associated risks, and cost and schedule is fundamental.

Orion flies in space while docked with a lunar lander in this NASA artist's rendering. (NASA)

Since 1960, The Aerospace Corporation has provided such objective assessments for national security space programs and recommendations for addressing potential problems and reducing risk. As its support for NASA programs has steadily grown over the years, Aerospace has increasingly called upon these capabilities to provide objective programmatic and technical assessments of NASA projects. In the programmatic area, Aerospace assists NASA in assessing cost and schedule at different points in project lifecycles to make important decisions about concept selection, designs, and implementation.

In the early phase of a project's lifecycle, Aerospace helps NASA select the best concept by assisting in the evaluation of proposals submitted in response to a NASA Announcement of Opportunity. Technical and cost analysis of notional approaches to implement future space telescope missions is an example of an Aerospace assessment to provide NASA early in the design cycle with insight into the mission's ability to achieve technical and programmatic goals. Aerospace also helps NASA make the best design choice, as it did in recommending the cost-effective RS-68 engine for the Ares V cargo-launch vehicle, prompting NASA to switch from its earlier decision to use the space shuttle main engine. Finally, Aerospace developed the Strategic Scenario and Contingency Modeling capability to support NASA in strategic planning and program portfolio analysis.

Programmatic Assessment Continuum

Aerospace has developed a suite of tools to assess the cost and schedule at different points within a project's lifecycle, based upon the available information. This continuum of tools provides a process to estimate the initial cost at the conceptual design stage through the preliminary system design and then to monitor and assess cost and schedule throughout the implementation of the mission. The process allows continual feedback to NASA to affect design or implementation changes.

In the conceptual design phase, the Complexity Based Risk Assessment (CoBRA) and Integrated Cost and Schedule Analysis Tool (ICSAT) are employed for early feasibility and mission-scoping studies. ICSAT, which Aerospace developed for NASA, estimates the cost, schedule, and budget profile for a full NASA mission based upon historical analogies, given a set of minimal mission and instrument requirements. The Aerospace-developed CoBRA model provides a correlation between the cost, schedule, and complexity of historical missions to identify appropriate cost and schedule budgets as a function of a complexity metric.

The programmatic assessment continuum was designed to address project executability throughout the NASA lifecycle. Aerospace has developed a suite of tools to assess the cost and schedule at different points within the project's lifecycle.

For evaluating projects in the formulation phase, Aerospace has pioneered the use of adjusted analogy-based, cost-estimating methods for both instrument and spacecraft estimates. Aerospace has also developed tools such as the Small Satellite Cost Model, which has been evolving continually since the early 1990s, as well as the Space-based Optical Sensor Cost Model to estimate satellite bus and scientific instrument cost for NASA missions. Aerospace uses a variety of methods, including analogy-based methods, which rely on historical information from similar past projects, to conduct independent cost estimates to ensure the robustness of the answer and to use as input to its cost-risk methodology. In 2004, Aerospace developed a process for performing independent schedule estimates that uses an analogy-based schedule assessment methodology similar to the one used when evaluating project costs, which permits greater insight into mission development times.

For evaluating projects in the implementation phase, earned-value-management probabilistic cost and schedule risk methods are employed. These methods assess the project's ability to implement according to its plan and provide valuable insight into problem areas and their effect on the project's ability to complete on time and on budget. Taken together, these methods provide a cost and schedule analysis continuum to address the assessment needs throughout a project's development lifecycle.

Deep Impact made history when it intercepted comet Temple 1 on July 4, 2005. It is now on a mission to fly by comet Hartley 2 on Oct. 11, 2010. As it cruises toward the comet, Deep Impact will observe five nearby stars with "transiting exosolar planets," named as such because the planet transits, or passes in front of, its star. (NASA/JPL/UMD. Artwork by Pat Rawlings)

Proposal Evaluation Support

Aerospace aids NASA in making decisions in the early phase of a project's lifecycle in evaluating submitted proposals. Its mix of independent, technical expertise across all spacecraft, launch vehicle, and ground systems disciplines—skills developed for Department of Defense acquisition support—makes Aerospace an ideal organization to assist NASA with the evaluation process. Over the last decade, Aerospace has performed independent assessments for the NASA Science Support Office at NASA Langley Research Center, which supports the NASA Science Mission Directorate in the acquisition of Earth and space science missions and instruments (see sidebar, NASA Awards).

Aerospace evaluates proposals for technical feasibility, risk posture, new technology development, cost and schedule implications, and other considerations. The assessment determines which proposals in response to a particular NASA Announcement of Opportunity are more likely to accomplish the science objectives within the planned resources.

Aerospace has used its programmatic and technical suite of tools in evaluating more than 500 full-mission and instrument-only proposals for NASA programs, including Discovery, New Frontiers, Mars Scout, Medium Explorer, Small Explorer, Earth System Science Pathfinder, New Millennium, and other programs dating back to the first Discovery evaluation in 1995. Three current operational missions—Deep Impact, MESSENGER, and Phoenix—were selected through this review process. The Deep Impact and MESSENGER missions were selected as part of the Discovery program in 1999, and the Phoenix mission was selected as part of the Mars Scout category within the Mars program in 2002.

Deep Impact successfully intercepted comet Temple 1 on July 4, 2005, and provided substantial insight into the internal composition of comets. Deep Impact is the first space mission to probe beneath the surface of a comet and reveal the secrets of its interior. The MESSENGER mission launched in 2004 is on its seven-year journey to orbit and map the planet Mercury. The Phoenix lander—launched in 2007 and successfully landed on the Mars North Pole on May 25, 2008—has embarked on the mission to determine whether Mars has ever harbored water.

A concept image shows the Ares V cargo launch vehicle. The envisaged heavy-lifting Ares V will be NASA's primary vessel for safe, reliable delivery of large-scale hardware to space. This includes the Altair lunar lander, materials for establishing a permanent moon base, and the vehicles and hardware needed to extend a human presence beyond Earth orbit. (NASA/MSFC)

Future Space Telescope Mission

Aerospace also supports NASA's Science Mission Directorate early in the design cycle to assess whether missions can achieve their technical and programmatic goals. For example, commissioned by NASA in November 2007, Aerospace in eight weeks completed a technical and cost analysis of several notional approaches to implement future medium-class space telescope missions capable of performing a broad array of science investigations, from extragalactic surveys to searches for exoplanets. This exercise was developed as part of an effort to improve early cost estimation for space missions.

Aerospace performed a technical concept validation and developed cost estimates for mission concepts, based on instrument complexity and basic system parameters such as size, weight, data rate, and data storage. Concepts ranged from 1–2 meter aperture sizes, with operations in the near infrared and visible region. A multidisciplinary team assessed technical design and representative cost and schedule, and identified, in particular, the cost drivers and payload differentiators. The science instruments require resources that define the instrument in terms of mass, power, data rate, and pointing control, as well as orbit requirements that, in turn, allow the resources of the spacecraft to be determined. The cost analysis highlighted the cost sensitivity to focal plane and aperture differences (also called collecting area whereby diameter is measured in meters) and other key optical parameters.

The results of the analyses provided NASA with an objective comparison for validating and scaling information to match technical content with budgetary resources. NASA commended Aerospace for the "thoroughness and comprehensiveness of the study in such a short turnaround time."

Lunar Transportation Architecture

Aerospace supports Constellation program confidence-level estimates and risk analysis efforts for the lunar program elements—including the Altair (lunar lander) project, mission operations, ground operations, and extravehicular activity project. An important aspect of the confidence-level estimate is the identification of areas that pose a high risk to the program in terms of technical challenges and/or cost and schedule overrun. Aerospace has assisted NASA with the identification of major risk drivers and communicated them to the project and program managers (see sidebar, NASA Engineering and Safety Center).

Aerospace also developed cost and risk assessments of several cargo launch vehicle (Ares-V) and lunar lander (Altair) configurations. The assessments were commissioned by NASA Exploration Systems Mission Directorate to provide an independent assessment of the expected cost and associated risks of these systems. These results will be used by NASA to validate internal assessments which are being used as part of the lunar campaign design process.

A Pratt & Whitney Rocketdyne RS-68 engine undergoing hot-fire testing during its developmental phase. A cluster of five RS-68 engines will power the Ares V. (NASA)

RS-68 Engine Assessment

Aerospace also aids NASA in assessing design choices for developmental vehicles and directly contributed to the design of the next American lunar launch vehicle. In response to President George W. Bush's announcement in January 2004 that the United States would return humans to the moon by 2020, NASA conducted the Exploration Systems Architecture Study. That study concluded that lunar launch requirements could best be met with a combination of a smaller, EELV-class crew launch vehicle, referred to as Ares I, and a larger, Saturn V-class cargo launch vehicle, referred to as Ares V. The new vehicle designs offer a solution for separating crew and cargo as specified by the Columbia Accident Investigation Board report, and are predicted to replace the space shuttle, which is scheduled to be retired in 2010. The new vehicles will provide support to the International Space Station and future solar system exploration missions.

As directed by Congress, NASA derived the design of both Ares rockets from existing space shuttle components in an attempt to minimize lifecycle cost, maximize workforce retention, and reduce development risk. In addition, the NASA Exploration Systems Architecture Study recommended using the space shuttle main engine—the most advanced, efficient, and expensive rocket engine ever built—for both the Ares I and Ares V rockets. Shortly after the study was completed, however, NASA's design analysis showed that using the shuttle main engine on the Ares I would be more difficult than originally envisioned, and NASA replaced it with the J-2X, a rocket engine derived from Apollo-era Saturn V hardware.

NASA asked Aerospace to reevaluate the impact of replacing the space shuttle main engine on the Ares V with the RS-68, a cost-effective engine that had been developed for the Delta IV launch vehicle. The Exploration Systems Architecture Study had advised against using the RS-68, pointing to the considerable additional cost, complexity, and development risk. It also suggested that the lower efficiency of the RS-68 would require substantially larger propellant tanks with the possibility that the Ares V might outgrow the vehicle assembly building at Kennedy Space Center. The Aerospace study was to determine whether using the RS-68 engines could offset the increased development and production costs of larger propellant tanks while continuing to meet the height requirements of the vehicle assembly building.

Aerospace showed that RS-68 based vehicles could be smaller than had been estimated by the architecture study because Ares V designs based on the shuttle main engine had used relatively inefficient trajectories to reduce the number of engines expended. The analysis leveraged the higher thrust of the RS-68 to increase the efficiency of the ascent trajectory, thereby offsetting the higher specific impulse of the space shuttle main engine. Aerospace also found that the cost of increasing the vehicle tank size was not as high as had been assumed, and that the cost savings from entirely eliminating the main engine infrastructure were substantial.

After independently verifying these findings, NASA chose to use the RS-68 for the Ares V, canceled its procurement of new space shuttle main engines, and began the process of shutting down the main engine production line. In a letter of commendation to the Aerospace team, NASA wrote that Aerospace efforts "provided NASA the rare opportunity to both increase performance and decrease costs." NASA estimates that the switch to the RS-68 will save from $3 billion to $6 billion through 2016, and additional savings of hundreds of millions of dollars per lunar mission are expected.

This artist's concept depicts NASA's Mars Phoenix Lander just before its 2008 touchdown on the arctic plains of Mars. Pulsed rocket engines control the spacecraft's speed during the final seconds of descent. (NASA/JPL)

Earned-Value Management

Aerospace uses earned-value management to provide NASA with important information on the progress of missions being carried out. Earned-value management is a specific, well-defined set of procedures used in program control to track expenditures and their relationship to the amount of work that has been accomplished. The earned-value data (the budgeted cost of work actually completed) provided by a project may be used to predict its estimate at completion, or expected total cost. A number of performance indices are used—cost performance, schedule performance, and schedule cost—which vary over time and can provide significantly different results.

Aerospace's cost-risk methodology for earned-value management uses these different performance indices to calculate the corresponding estimates at completion for each work breakdown structure (WBS) element cost to develop a total system estimate at completion cost distribution. The WBS is a list of everything that has to be paid for to bring a system to its full operational capability. The Aerospace process also uses the earned-value management data to perform an earned-schedule assessment to predict the schedule at completion. This includes a Monte Carlo simulation of the schedule and a probability assessment of the launch readiness date for a system.

Aerospace has been using this methodology to support the NASA Mars Exploration program office since 2007 to provide continual insight into the development of the Mars Science Laboratory (MSL) elements. These analyses have provided NASA with the ability to target preventative measures and recovery activities on key development and test areas.

MSL is a rover that will assess whether Mars ever was, or is still today, an environment able to support microbial life. MSL will travel farther, carry more instruments, and sample more rocks and soil than its predecessors. (See the back page of this Crosslink for more on MSL.)

Strategic "Sand Chart" Analysis

Aerospace research into the reasons for NASA cost and schedule growth led to the development of the Strategic Scenario and Contingency Modeling capability, referred to at NASA as the "Sand Chart" tool. Aerospace developed this capability to support NASA in strategic planning and program portfolio analysis.

Starting with a program architecture (defined as a portfolio of either loosely or tightly coupled projects) and an available program funding wedge, the Sand Chart tool inputs probabilistic cost risk assessments for each of the program's elements or projects and simulates the dynamics of the interaction of the cost and schedule growth for project elements. Real cost and schedule overrun data from NASA projects and the interrelationships of these projects are used to inform the model algorithms.

The Mars Science Laboratory (MSL) will carry a laser for vaporizing a thin layer from the surface of a rock and analyzing the elemental composition of the underlying materials. MSL will be able to collect rock and soil samples and distribute them to onboard test chambers for chemical analysis. It's designed to have a suite of scientific instruments for identifying organic compounds such as proteins, amino acids, and other acids and bases that attach themselves to carbon backbones and are essential to life. (NASA/JPL)

This top-level risk and sensitivity framework complements traditional budget risk analysis, which typically only captures risks owing to cost model and input uncertainties, not discrete outside influences. Such influences can have more dramatic impacts on programs. This capability has given NASA a better understanding of the primary internal and external influences on its programs for the next several decades. The analytic framework includes future planned decision points and a range of unplanned events and provides for comparing architecture alternatives to support strategic decisions. The result is a program schedule and program plan laid out against the budget that is robust (and achievable) to a range of likely "what-if?" scenarios.

NASA applies this process to many areas, such as how soon the first lunar landing will take place and how many science missions can be flown by a certain date within the expected budget. Moreover, the analysis has already affected program direction several times and is being used to assist with budget planning. This capability, in conjunction with the other tools and analyses described in this article, allows Aerospace to provide strategic guidance and decision support at the highest levels of NASA's management.

Acknowledgments

The authors thank Dave Bearden, Torrey Radcliffe, Inki Min, Marcus Shaw, Marcus Lobbia, Dean Bucher, and Eric Breckheimer for their significant contributions to this article. The Aerospace Corporation efforts presented in this article represent work by many experts from a variety of departments and organizations over many years. NASA has also publicly recognized a number of these people for their efforts.