Estimating Probable System Cost

Estimating Probable System Cost

Stephen A. Book

Basing a system cost estimate on past systems can be tricky. Analysts who do this find that the sum of the most likely costs of the elements of a space system in development does not equal the most likely cost of the entire system. Using probability distributions to treat cost estimation as a statistical process can provide estimates that are much more meaningful.

In estimating the cost of a proposed space system, cost analysts follow the adage "What's past is prologue." They use costs of existing systems to develop a cost-estimating relationship (CER), which can help predict the cost of a new system. The foundation of modern cost analysis, the CER is usually expressed as a linear or curvilinear statistical regression equation that predicts cost (the dependent variable) as a function of one or more cost drivers (independent variables). However, the CER tells only part of the story. At the beginning of a cost-estimating task, the analyst identifies the work-breakdown structure (WBS), a list of everything that has to be paid for to bring a system to its full operational capability. A space system's WBS includes high-level categories such as research, development, and testing; operations, maintenance, and support; production; and launch. Lower levels of the structure include software modules, electronics boxes, and other components. In addition to CERs, the analyst bases estimates on costs of items already developed and produced, on vendor price quotes for off-the-shelf items, and on any other available information that can be used to assign a dollar value to items in the WBS.

Until recently, sponsors and managers of space systems have expected cost analysts to provide best estimates of the costs of various options at each project milestone and decision point, from the initial trade-study stage to source selection, right on to project completion. Unfortunately, "best estimate" has never been precisely defined. For example, is it the most likely (most probable) cost, the 50-percent confidence level cost (the dollar figure that is equally likely to be underrun or overrun), or the average cost? It wasn't clear what useful information about system cost this best estimate was conveying, so the figure proved inadequate for comparing competing options, as well as for planning system budgets. This unsatisfactory situation led the Department of Defense (DOD) to issue formal guidance on how system costs should be expressed and what the terminology should mean.

The typical approach to obtaining a best estimate is to view each WBS item's estimated cost as its most likely cost and then roll up (sum) those estimates to arrive at a total system cost. Because high-end risks outweigh low-end uncertainties, however, a roll-up estimate calculated this way tends to underestimate actual cost by a wide margin, leading to cost overruns attributable purely to the mathematics of the roll-up procedure. In fact, formal mathematical theory confirms that the sum of the most likely WBS-item costs is substantially less than the most likely total cost. Experience shows that assigning a confidence level of 30 percent or less to a roll-up estimate is not overly pessimistic. While generally thought to be a new phenomenon specific to DOD, the difficulties associated with cost-estimating uncertainty were recognized in France as early as 1952.

A WBS is a hierarchical list of all items that must be paid for to bring a system to its full operational capability. A space system's WBS includes high-level categories such as launch vehicle, as well as low-level items such as engines and software.

The solution to the problem is to treat the cost-estimating process statistically, a technique known as cost-risk analysis. Even the use of the term "most likely" indicates a statistical situation, because it implies that other, less likely estimates exist. Probability distributions are established to model the cost of each WBS item. (A probability distribution contains the possible values a random variable could take on, as well as the probability of it taking on each one of those values.) Then, correlations among these distributions are estimated, and the distributions are summed statistically, typically by Monte Carlo sampling. The result is a probability distribution of total system cost, from which one can obtain meaningful estimates of the median (50-percent confidence level), 70th percentile (70-percent confidence level), and other relevant quantities. The cost-risk-analysis approach yields a mathematically correct most likely cost (one that is precisely defined, along with its level of confidence), as well as costs for all percentiles. Estimates at the 50-percent and 70-percent confidence levels are much more valuable to decision makers in setting program budgets than an essentially meaningless "most likely" estimate. In the early 1990s The Aerospace Corporation developed many of the mathematical procedures now applied to estimate project costs statistically.

Once the budget has been established and the system has been in development for a number of months, earned-value data become available. A measure of how much work has been accomplished on a project, earned-value data are typically collected by the contractor for use in comparing actual expenditures on scheduled tasks with the amounts budgeted for them. Over a specific time period, usually a month or three months, and cumulatively since the beginning of development, an earned-value management system tracks and compares three kinds of financial information associated with each WBS item: the budgeted cost of work scheduled to be done on the item during the period (the estimated cost), the budgeted cost of work actually completed on that item during that time (the earned value), and the actual cost incurred, per billing, for the contractor's work done on the item during that time. Earned-value data can be used to forecast an estimate at completion at any point in the program.

Measuring the Quality of a Cost-Estimating Relationship

How does a cost analyst derive a CER for a WBS item? Let's say the item under consideration is a satellite's solar-array panels. Cost and technical data on solar arrays that have already been produced must be organized into a database. For each array, cost is tracked against appropriate technical characteristics such as weight, area, and storage capacity. Algebraic relationships between cost and these potential cost drivers are compared to determine which relationship is the optimal predictor of solar-array cost. It is not a priori obvious which criteria are the best to use for assessing CER appropriateness, nor will all cost analysts agree on the best criteria. Three statistical criteria, however, lie at or near the top of almost everyone's list:

percentage standard error of predictions made by the CER of values in the database: root-mean-square of all percentage errors made in estimating values of the database using the relationship (a one-sigma number that can be used to bound actual cost within an interval surrounding the estimate with some degree of confidence)
net percentage bias of predictions of the values in the database: algebraic sum, including positives and negatives, of all percentage errors made in estimating values of the database using the CER (a measure of how well percentage overestimates and underestimates of database actual costs are balanced)
correlation between estimates and actual costs (CER-based predictions and cost values in the database): If the relationship were a perfect predictor of the actual cost values of elements of the database, a plot of estimates against the actual costs would follow a 45-degree line quite closely (correlation, a statistical measure of the extent of linearity in a relationship between two quantities, would be high if estimates tracked actual values but low if they did not).

Standard error is better expressed in percentage terms than in dollars. Using percentage to express standard error in cost-estimating offers stability of meaning across a wide range of programs, time periods, and situations. An error of 40 percent, for example, retains its meaning whether the analyst is estimating a $10,000 component or a $10 billion program. Conversely, a $59,425 error is huge when reported in connection with a $10,000 component, but insignificant with respect to a $10 billion program. Even in less extreme cases, a standard error expressed in dollars often makes a CER virtually unusable at the low end of its data range, where relative magnitudes of the estimate and its standard error are inconsistent.

Similarly, in the case of bias, a dollar-valued expression is not as informative as an expression in terms of percentage of the estimate, because a particular dollar amount of bias would not have the same impact on all values in the database (see sidebar, Statistics at a Glance).

Cost-Risk Analysis

"Cost-risk analysis" is the term used by cost analysts for any estimating method that treats WBS-item costs and total-system costs as random variables rather than deterministically derived numbers. The term implies that, for any deterministic estimate, there is some degree of risk that the system will be unable to be delivered or meet its stated objectives at that particular funding level. Cost-risk analysis recognizes that a mathematical probability of success is associated with each deterministic cost estimate (see sidebar, Cost Estimating at a Glance).

Are costs really random? The formal framework for working with a range of possible numeric values is the probability distribution, the mathematical signature of a random variable. Modeling costs as random variables does not imply they are random. It reflects how an item's cost results from a large number of very small influences, whose individual contributions we cannot investigate in enough detail to precisely calculate the total cost. It is more efficient to recognize that virtually all contributors to cost are uncertain and to find a way to assign probabilities to various possible ranges.

Consider coin tossing. In theory, if we knew all the physics involved, we could predict with certainty whether a coin would land heads or tails; however, the influences acting on the coin are too complex for us to understand in enough detail to calculate the parameters of the coin's motion. So, instead, we bet that the uncertainties will average out in such a way that the coin will land heads half the time and tails the other half. It is more efficient to model the physical process of coin tossing—which is in fact deterministic—as if it were a random statistical process and to assign probabilities of 0.50 to each of the possible outcomes. System cost can be similarly represented as a random variable rather than a fixed number, because cost is composed of many very small pieces, whose individual contributions to the whole cannot be specified in sufficient detail for precise estimation of the whole.

Standard accounting technique considers total system cost to be the sum of the costs of all WBS items, so it first requires us to estimate the most likely cost (the mode, the most frequently occurring value) of each item and then to sum those costs. While this procedure seems reasonable, the roll-up is almost sure to be very different from the actual most likely value of the total cost. Statistical theory shows that the sum of modes does not generally equal the mode of the sum. Because of the preponderance of high-end risks (with more probability lying above the best estimate than below it), most cost probability distributions are not symmetric about their modes. As a result, the sum of the modes is usually considerably smaller than the mode of the sum.

The practical impact of this is that estimates obtained by totaling most likely costs of WBS items tend to significantly underestimate actual cost. Examples are available to illustrate what kinds of errors might occur in typical cases if these mathematical peculiarities are ignored. They show that a roll-up estimate typically has a probability of 30 percent or less of being sufficient to fund the program. Here as elsewhere, short-cutting proper statistical procedures leads to erratic, unpredictable results.

Cost Correlation Between Items

Correlation among WBS-item cost distributions contributes significantly to the uncertainty of the total system cost; this has been realized since the probabilistic approach to cost analysis became de rigueur over the past decade in response to DOD-issued requests for proposals. Understanding the degree of uncertainty in an estimate is a necessary aspect of cost analysis, one that is paralleled by representing system cost as a random variable for the purpose of appropriately modeling uncertainty.

This graph illustrates the importance of working with the numeric correlations between WBS items. Assuming these correlations to be zero causes a detrimental effect on the estimation of total-cost uncertainty. Shown is the percentage by which the sigma value (standard deviation) of the total-cost distribution is underestimated, assuming WBS interelement correlations to be zero instead of the actual value (usually represented by , the Greek letter rho). The horizontal axis tracks , and the vertical axis tracks the percentage by which the total-cost sigma value is, for each nonzero correlation value, underestimated if the correlations are instead assumed to be zero. Each curve is keyed to a unique value of n, the number of elements in a roll-up. As the four curves show, the percent by which sigma is underestimated also depends on the number of WBS items for which the pairwise correlations are incorrectly assumed to be zero. For example, if n = 30 WBS items, and all correlations between WBS items () are 0.2, but the estimator assumes they are all zero, the total-cost sigma values would be underestimated by about 60 percent. (This is meant to be a generic illustration and therefore is only approximately true in any specific case. It has been assumed that the sigma values for the WBS items are the same throughout the entire structure.)

Because risks faced in working on different WBS items are often correlated, ignoring correlation in statistical computations makes the spread of the cost distribution narrower than it should be. Failing to account for correlation therefore deceives the analyst by making an estimate appear less uncertain than it really is. The most universal statistical descriptor of a random variable's degree of uncertainty (spread) is its standard deviation (sigma value), and pairwise correlations between random variables are significant contributors to the magnitude of the sigma value of their sum. This makes correlation between program-item costs a critical factor in the estimation of total system cost uncertainty.

Cost estimating would be much simpler if there were no interelement cost correlations; unfortunately, this is not the case. Consider the example of a space system. At the highest level of the WBS, there are three elements: space-resident satellites, a launch system, and a ground-based control and data-analysis system. The respective costs X_S, X_L, and X_G may be positively correlated for several reasons. An increase in size, weight, and number of satellites to be placed in orbit results in an increase in launch costs, either through the number of launches required or the needed capability of the individual launch vehicles. An increase in number and data-gathering capability of the satellites forces an increase in ground-operations costs, either through the complexity of the tasking and control system software or the number and size of ground-station facilities.

On the other hand, X_S, X_L, and X_G may be negatively correlated for different reasons. Reducing the complexity of onboard satellite software and communications hardware may increase ground costs by complicating the ground software while, at the same time, decreasing launch costs as a result of reduction in size of on-orbit hardware.

Sample triangular probability distribution of WBS-item cost, based on estimation of optimistic, most likely, and worst-case costs. All statistical properties of the triangular distribution are determined by the lowest possible cost L, the most likely cost M, and highest possible cost H.

Further down into the WBS, costs are more highly correlated because they correspond to specific items that physically occupy adjacent locations within the satellites or ground stations or to software packages that operate specific pieces of hardware. In any particular case, the actual interelement correlations have to be estimated along with the element costs themselves.

While it may be difficult to justify use of a specific numeric value to represent the correlation between two WBS-item costs, it is important to avoid the temptation to omit the correlation altogether when a precise value for it cannot be established. Such an omission will set the correlation in question to the exact value of zero, whereas positive values of the correlation coefficient tend to widen the total-cost probability distribution and thus increase the gap between a specific cost percentile (e.g., 70 percent) and the best-estimate cost. Therefore, using reasonable nonzero values, such as 0.2 or 0.3, generally leads to a more realistic representation of total-cost uncertainty. Having been the first organization to vigorously advocate in Washington in the early 1990s that account be taken of the significant impact that correlation exerts on cost estimating, Aerospace has developed most of the practical methods currently in use for dealing with correlation-related issues.

Obtaining Cost Percentiles

Cost-risk analysis comprises a series of engineering assessments and mathematical techniques whose joint goal is to measure the degree of confidence attached to any particular estimate of system cost. A three-step procedure built upon results of a technical-risk study typically forms the cost-risk analysis. First, an engineering assessment of the technologies involved in each subsystem leads to probability distributions of subsystem costs. Second, these subsystem cost distributions are correlated and combined to generate a cumulative distribution of total system cost. Finally, once the cumulative distribution has been established, the 50th, 70th, 90th, and other cost percentiles can be read off the graph.

Once the cost analyst has established probability distributions of individual WBS-item costs, then Monte Carlo sampling is carried out from each subsystem's distribution. The random numbers thus generated are combined in a logical way to produce a representation of the cumulative distribution of total-system cost.

Based on engineering assessments, the cost-estimation process is carried out by assigning low, best, and high cost estimates to each item in the WBS. These three estimates define a triangular probability distribution. All statistical properties of this distribution are uniquely determined by three parameters: the lowest possible cost L, the most likely cost M, and the highest possible cost H. The low estimate specifies subsystem cost under the most optimistic assumptions concerning development and production capabilities. The most likely estimate is typically derived from the output of a CER-based cost model or other appropriate estimating procedure such as analogy or engineering buildup. The high-end cost encompasses the impacts of all technical risks faced in developing and producing the subsystem. Translating qualitative and quantitative assessments of technical risk into dollars to determine a realistic high-end cost typically requires extensive interactions of cost analysts with engineers knowledgeable in the technical state of the art.

System-design concepts usually contain physical descriptions or lists of engineering requirements that may be translatable into dollars. Such translations may be derived from knowledge of technical precedents on which the concept is based. Alternatively, they may be derived from the fact that such precedents are lacking or have not been successfully pursued in the past, despite expenditures of known amounts of funds. Technical complexity of unproved methods for implementation are assessed using scales of increasing difficulty, complexity, or uncertainty analogous to related events in the historical record. These technical-risk measurements may then be translated into cost risks, based on analogous cost experience or state-of-the-art relationships among cost, technical difficulty, and pace of development. Management and control of costs may then be implemented in accordance with a realistic understanding of the primary source of risk, namely technical difficulty.

After probability distributions of individual WBS-item costs have been established, the next step is Monte Carlo random sampling from each subsystem's distribution and combining these random numbers in a logical way to produce a representation of the cumulative distribution of total system cost. The ultimate objective of cost-risk analysis, the ability to read off percentiles of total system cost, is thus achieved.

Trade Studies and Source Selections

For programs in progress, probabilistic information allows budget planning to be based on the likelihood that any proposed dollar amount will be adequate to fund the program. Prior to formal program initiation, trade studies are typically undertaken to find out whether a certain type of system is feasible from the operational and cost points of view. Additionally, source selections are conducted to evaluate system approaches to a problem proposed by different contractors (under acquisition reform, the program for improving and accelerating government contracting and procurement while reducing costs, the approaches may possibly even meet different sets of requirements).

Sample lognormal probability distribution, a distribution in which the natural log of a random variable is normally distributed. Unique mathematical characteristics of the triangular and lognormal probability distributions make them both especially applicable to cost analysis at the trade-study and source-selection stages. A random variable has a lognormal probability distribution if its natural logarithm has a normal probability distribution; a normal distribution is the familiar bell curve, a continuous distribution that is symmetric about its mean. The asymmetrical lognormal distribution, a good model for the statistical sum of a number of triangular distributions, is an excellent choice for representing total-cost distributions of systems that are to be compared on the basis of their relative cost. This is because the ratio of two independent lognormals is itself lognormally distributed.

Timeliness and simplicity are key requirements of analyses undertaken in support of trade studies and source selections because not much technical detail is available about the system under study during either phase. In both cases, a decision has to be made by someone who has a very limited factual database. For trade studies and source selections, probabilistic information allows candidate systems to be compared on a level playing field; the go-ahead decision (in a trade study) or contract award (in a source selection) can be made on the basis of, say, the 50th percentile cost of each candidate.

But a nagging question remains: "What if System A, the lower-cost option at the 50-percent confidence level, faces risk issues that make its 70th-percentile cost higher than that of System B?" In other words, System B would be the lower-cost option if the cost comparison were made at the 70-percent confidence level, while System A would be the lower-cost option if the cost comparison were made at the 50-percent level. In this classic situation, the decision maker has to choose between a low-cost, high-risk option and a high-cost, low-risk option. To take account of all possible risk scenarios, the decision maker can make use of all cost percentiles simultaneously, namely the entire cost probability distribution of each candidate system (which reflects the candidate system's entire risk profile), not simply the 50-percent or the 70-percent confidence cost. As it turns out, the expression of system cost in terms of a probability distribution makes it possible to estimate the probability that System A will be less costly than System B, and that probability can be part of the basis on which the decision is made.

Learning Rates

Standard cost-estimating practice involves the application of a cost-improvement factor, or learning rate, to account for management, engineering, and/or production improvements that save money as successive units are produced. Lack of credible analogous or applicable historical data, however, makes it difficult or even impossible to determine in advance exactly what an accurate learning rate will be in any particular estimating context.

Cost-estimating practice involves the application of a learning rate to account for improvements that save money as successive units are produced. This graph illustrates how cost can vary at different learning rates.

Nevertheless, the estimator's choice of learning rate exerts a major impact on the estimate of the total spending profile of a large production program. Even if nonrecurring and first-unit production costs are estimated precisely, small variations in the learning rate will substantially outweigh all other contributions to uncertainty in the total system estimate.

This is especially true in cases of large-quantity procurements, such as aircraft, launch vehicles, or proposed constellations of numerous satellites. In the case of 50 units, for example, the average-unit-cost estimate will be 46.6 percent lower at an 85-percent learning rate, compared with what it would be at a 95-percent learning rate. For 200 units, the estimate will be 57.3 percent lower at an 85-percent learning rate versus a 95-percent learning rate. In the case of 5000 units, the estimate will be 74.5 percent lower at an 85-percent versus a 95-percent learning rate. The learning rate should therefore be treated as another source of cost risk, with optimistic, most likely, and pessimistic learning rates factored into the total system cost probability distribution.

Allocating Risk Dollars for Reserve

Because users of common estimating methods often underestimate actual project cost, a management reserve fund is a smart idea. This fund is put in place to help overcome unanticipated contingencies that may deplete the budget prior to project completion. Percentiles of the cost probability distribution can serve as guidelines for the magnitude of the appropriate management reserve. Suppose that the number submitted as the best estimate falls at the 40th-percentile level of the cost probability distribution. Depending on the importance of the project, a prudent reserve might consist of the funding required to bring the total available dollars to at least the 50th, 70th, or even 90th percentile.

NASA technology readiness level (TRL) scale. This tool provides information for determining the worst-case high-end cost of a WBS item. For each spacecraft, aircraft, or payload subsystem under study, an engineer assigns one of the TRL indexes to that subsystem, then compares that index with the indexes of each item in the database that supports the cost estimate. The engineer can then derive the triangular distribution of cost for that subsystem from the relationship between the average TRL index of the database and the TRL index of the subsystem under study. Each time a new program's cost is estimated, its TRL level will likely be higher than what it had been, if progress is being made. When a new program is at a lower level than the database, its cost will be more uncertain and its cost probability distribution will tend to range far from the estimate derived from the database. As work proceeds, though, the new program's TRL level should eventually exceed the average level of the database. Then its cost will be less uncertain, so its cost probability will be concentrated somewhat closer to the database estimate.

Risk dollars in the management reserve pool may sometimes constitute a large percentage of the budgeted best-estimate funding base. Funding agencies are reluctant to set aside such large amounts of money for management reserve, believing that "slush funds" lead to waste and slack management. It is therefore advisable to justify such requests by providing a logical allocation of the requested risk dollars among the various project elements. A specific WBS-based cost-risk analysis can profile a probable need for additional monies beyond those included in the best estimate.

Because a WBS item's need for risk dollars arises out of uncertainty in the cost of that item, a quantitative description of that need must serve as the mathematical basis of the risk-dollar computation. In general, the more uncertain the cost, the more risk dollars will be needed to cover a reasonable probability (e.g., 0.70) of being able to complete that element of the system. Items whose cost distributions have relatively high probabilities of exceeding their own most likely estimates will need more risk dollars. Methods were developed at Aerospace to allocate management reserve properly, taking into account not only the skewness of each item's probability distribution, but also correlations between the items, so that management reserve dollars will not be assigned to do double duty.

Estimates at Completion Based on Earned Values

Once a program is under way, program managers must monitor how work is being done and money is being spent. Earned-value management is a specific, well-defined set of procedures used in program control to track expenditures and their relationship to the amount of work that has been accomplished. Earned-value-management documentation compares outflow of program funds with completion of various work packages against which the funds have been budgeted. This comparison, used properly, allows program-control personnel to quickly spot overruns and possible schedule discrepancies. In addition, earned-value-management data are used to calculate estimates at completion at any stage in the program. Despite the wealth of data that earned-value-management systems bring to bear on the estimating process, they have not been able to circumvent the statistical nature of cost that, for at least the past decade, has been the driving force behind the development and application of cost-risk analysis.

The two main quantities formally tracked by earned-value-management systems are cost variance and schedule variance. Cost variance is the difference between the amount of money budgeted for work actually completed (or completed over some time period, such as a month) and the amount of money actually spent to do that work, regardless of how much work was supposed to get done during that period. Schedule variance is the difference between the budgeted cost of the work completed (or completed over some period) and the budgeted cost of the work planned for that time period, regardless of the amount actually expended.

Mathematically related to the cost and schedule variances, although calculated slightly differently, are two other quantities: the cost-performance index and the schedule-performance index. The cost-performance index is a measure of the efficiency at which dollars are being spent on the project; for example, a cost-performance index of 0.90 means 90 cents worth of work is getting done for every dollar spent. The schedule-performance index measures the rate at which work is being completed; a schedule-performance index of 0.90 means 90 percent of the work is getting done that is supposed to be done during the time period in question.

card shuffle Focusing on the problem of using earned-value data to calculate estimates at completion, Professor D. S. Christensen of Southern Utah University tracked the historical performance of a number of common methods of making estimates-at-completion calculations. His research led him to conclude that final program cost almost always falls between estimates at completion based on two earned-value-derived indexes, the cost-performance index and the schedule-cost index. (The latter is the product of the cost-performance index and the schedule-performance index.) Furthermore, he found that actual program final cost is generally closer to the estimates at completion based on the schedule-cost index, which he refers to as a ceiling to the final cost. He calls the estimates at completion based on the cost-performance index the floor to the actual final cost, based on his conclusions that program cost performance rarely improves as the program proceeds to its completion.

The results of Christensen's research, namely the clear historical precedents for estimating a floor and ceiling to program final cost, were recently applied by analysts at Aerospace to construct a statistical approach to computing estimates at completion. Such an approach allows us to associate levels of confidence with various estimates at completion, to distinguish the dollar value of a best estimate at completion from risk dollars or management reserve, and to identify WBS items that are most likely to require an infusion of risk dollars. While this kind of information is usually inferred from a detailed technical risk analysis, it can also be derived from earned-value data.

Summary

dice toss At several stages in the system engineering process, it is necessary to conduct a cost analysis to assess the likely magnitude of program funding requirements (see illustration). The cost-analysis process begins at the trade-study stage with an applicable set of CERs statistically derived from historical cost data. After specific hardware and software designs have been formally adopted, it may be possible to base estimates on components that were previously developed and produced, vendor price quotes for existing off-the-shelf items, or other appropriate data that can enable dollar values to be assigned to all items to be paid for. These information sources typically allow the analyst to estimate the most likely cost of each item, but because many technical risk issues are often present, the sum of the items' most likely costs is usually significantly below the total system's most likely cost. This unfortunate situation led in the early 1990s to the development of the subfield of cost-risk analysis, a way of looking at cost through the lens of probability and statistics.

Once one accepts the idea of evaluating costs probabilistically, one is locked into the need to estimate correlations between the cost impacts of various technical risks because correlation is a significant driver of total-cost uncertainty. However, while cost-risk analysis makes demands upon the cost estimator, it also provides benefits to program management, particularly when it comes to recommending a prudent management reserve. Having in hand a probability distribution of total program cost, rather than just a single best estimate, program management can propose, for example, that the basic cost estimate be budgeted at the 50-percent confidence level, but that sufficient management reserve be included to bring the success probability up to 70 percent.

Finally, as the program progresses, earned-value data on expenditures and work accomplished provide program management with the ability not only to maintain current knowledge of cost overruns, but also to estimate cost at completion from inside the program itself rather than by statistical inference from historical information on other programs.