CEBoK Module 09 Cost and Schedule Risk Analysis
-
Upload
khangminh22 -
Category
Documents
-
view
0 -
download
0
Transcript of CEBoK Module 09 Cost and Schedule Risk Analysis
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 1
Cost and Schedule Risk Analysis Basic
How to adjust your estimate for uncertainty and historical cost and schedule growth
ldquoAs we know There are known knowns There are things we know we know We also know There are known unknowns That is to say
We know there are some things We do not know But there are also unknown unknowns The ones we dont know We dont knowrdquo
- ldquoThe Unknownrdquo from Pieces of Intelligence The Existential Poetry of DonaldRumsfeld Hart Seely 2003 [DoD news briefing 02122002]
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Acknowledgmentsbull ICEAA is indebted to TASC Inc for the
development and maintenance of theCost Estimating Body of Knowledge (CEBoKreg)ndash ICEAA is also indebted to Technomics Inc for the
independent review and maintenance of CEBoKreg
bull ICEAA is also indebted to the following individuals who have made significant contributions to the development review and maintenance of CostPROF and CEBoK reg
bull Module 9 Cost and Schedule Risk Analysisndash Lead authors Richard L Coleman Jessica R Summerville
Eric R Druker Richard C Leendash Senior reviewers Richard L Coleman Michael F Jeffers Kevin
Cincotta Fred K Blackburnndash Reviewer Bethia L Cullisndash Managing editor Peter J Braxton
2Unit III - Module 9
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 3
Unit IndexUnit I ndash Cost EstimatingUnit II ndash Cost Analysis TechniquesUnit III ndash Analytical Methods
6 Basic Data Analysis Principles7 Learning Curve Analysis8 Regression Analysis9 Cost and Schedule Risk Analysis10Probability and Statistics
Unit IV ndash Specialized CostingUnit V ndash Management Applications
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 4
Risk Overviewbull Key Ideas
ndash Risk bias (accuracy)ndash Uncertainty (precision)ndash Cost realismndash Risk vs Sensitivityndash Inputs vs Outputs Risk
bull Practical Applicationsndash Probabilistic Cost Estimates
bull S-Curvesbull Budgeting to Percentiles
ndash Risk Scoring and Mapping
bull Analytical Constructsndash Probability Distributions for Riskndash Percentilesndash Prediction Intervals (PI)ndash Correlation
bull Related Topicsndash Data Collection for
Risk Analysisndash Monte Carlo Simulationndash Risk Managementndash Schedule Analysis
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Agendabull Cost Risk Analysis
ndash Introductionndash Distributions
bull Fitting distributions to historical databull Measuring technical uncertaintybull Measuring estimating uncertainty
ndash Aggregating riskbull Simulationbull Method of momentsbull Correlation
ndash Risk measuresndash Risk allocationndash Joint confidence level analysisndash Practical exercises
bull Comparing simulation with method of moments
Unit III - Module 9 5
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Acknowledgmentsbull ICEAA is indebted to TASC Inc for the
development and maintenance of theCost Estimating Body of Knowledge (CEBoKreg)ndash ICEAA is also indebted to Technomics Inc for the
independent review and maintenance of CEBoKreg
bull ICEAA is also indebted to the following individuals who have made significant contributions to the development review and maintenance of CostPROF and CEBoK reg
bull Module 9 Cost and Schedule Risk Analysisndash Lead authors Richard L Coleman Jessica R Summerville
Eric R Druker Richard C Leendash Senior reviewers Richard L Coleman Michael F Jeffers Kevin
Cincotta Fred K Blackburnndash Reviewer Bethia L Cullisndash Managing editor Peter J Braxton
2Unit III - Module 9
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 3
Unit IndexUnit I ndash Cost EstimatingUnit II ndash Cost Analysis TechniquesUnit III ndash Analytical Methods
6 Basic Data Analysis Principles7 Learning Curve Analysis8 Regression Analysis9 Cost and Schedule Risk Analysis10Probability and Statistics
Unit IV ndash Specialized CostingUnit V ndash Management Applications
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 4
Risk Overviewbull Key Ideas
ndash Risk bias (accuracy)ndash Uncertainty (precision)ndash Cost realismndash Risk vs Sensitivityndash Inputs vs Outputs Risk
bull Practical Applicationsndash Probabilistic Cost Estimates
bull S-Curvesbull Budgeting to Percentiles
ndash Risk Scoring and Mapping
bull Analytical Constructsndash Probability Distributions for Riskndash Percentilesndash Prediction Intervals (PI)ndash Correlation
bull Related Topicsndash Data Collection for
Risk Analysisndash Monte Carlo Simulationndash Risk Managementndash Schedule Analysis
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Agendabull Cost Risk Analysis
ndash Introductionndash Distributions
bull Fitting distributions to historical databull Measuring technical uncertaintybull Measuring estimating uncertainty
ndash Aggregating riskbull Simulationbull Method of momentsbull Correlation
ndash Risk measuresndash Risk allocationndash Joint confidence level analysisndash Practical exercises
bull Comparing simulation with method of moments
Unit III - Module 9 5
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 3
Unit IndexUnit I ndash Cost EstimatingUnit II ndash Cost Analysis TechniquesUnit III ndash Analytical Methods
6 Basic Data Analysis Principles7 Learning Curve Analysis8 Regression Analysis9 Cost and Schedule Risk Analysis10Probability and Statistics
Unit IV ndash Specialized CostingUnit V ndash Management Applications
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 4
Risk Overviewbull Key Ideas
ndash Risk bias (accuracy)ndash Uncertainty (precision)ndash Cost realismndash Risk vs Sensitivityndash Inputs vs Outputs Risk
bull Practical Applicationsndash Probabilistic Cost Estimates
bull S-Curvesbull Budgeting to Percentiles
ndash Risk Scoring and Mapping
bull Analytical Constructsndash Probability Distributions for Riskndash Percentilesndash Prediction Intervals (PI)ndash Correlation
bull Related Topicsndash Data Collection for
Risk Analysisndash Monte Carlo Simulationndash Risk Managementndash Schedule Analysis
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Agendabull Cost Risk Analysis
ndash Introductionndash Distributions
bull Fitting distributions to historical databull Measuring technical uncertaintybull Measuring estimating uncertainty
ndash Aggregating riskbull Simulationbull Method of momentsbull Correlation
ndash Risk measuresndash Risk allocationndash Joint confidence level analysisndash Practical exercises
bull Comparing simulation with method of moments
Unit III - Module 9 5
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 4
Risk Overviewbull Key Ideas
ndash Risk bias (accuracy)ndash Uncertainty (precision)ndash Cost realismndash Risk vs Sensitivityndash Inputs vs Outputs Risk
bull Practical Applicationsndash Probabilistic Cost Estimates
bull S-Curvesbull Budgeting to Percentiles
ndash Risk Scoring and Mapping
bull Analytical Constructsndash Probability Distributions for Riskndash Percentilesndash Prediction Intervals (PI)ndash Correlation
bull Related Topicsndash Data Collection for
Risk Analysisndash Monte Carlo Simulationndash Risk Managementndash Schedule Analysis
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Agendabull Cost Risk Analysis
ndash Introductionndash Distributions
bull Fitting distributions to historical databull Measuring technical uncertaintybull Measuring estimating uncertainty
ndash Aggregating riskbull Simulationbull Method of momentsbull Correlation
ndash Risk measuresndash Risk allocationndash Joint confidence level analysisndash Practical exercises
bull Comparing simulation with method of moments
Unit III - Module 9 5
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Agendabull Cost Risk Analysis
ndash Introductionndash Distributions
bull Fitting distributions to historical databull Measuring technical uncertaintybull Measuring estimating uncertainty
ndash Aggregating riskbull Simulationbull Method of momentsbull Correlation
ndash Risk measuresndash Risk allocationndash Joint confidence level analysisndash Practical exercises
bull Comparing simulation with method of moments
Unit III - Module 9 5
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 6
Introductionbull Cost estimates in many cases project many years in the future
ndash Cost estimates are inherently uncertain regardless of whether risk is included (modeled)
bull Sources of uncertainty include (but are not limited to)ndash Scope and requirements (what is being estimated)ndash Technology readinessndash Manufacturing readinessndash Inflationndash Model uncertainty
bull The cost analysis profession recognizes the importance of risk and has incorporated risk analysis as an integral part of the estimating process
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Analysis Termsbull Risk is the chance of uncertainty or loss
bull Uncertainty is the indefiniteness about the outcome of a situationndash Includes both favorable and unfavorable events
bull Cost Risk is a measure of the chance that due to unfavorable events the planned or budgeted cost of a project will be exceeded
bull Cost Uncertainty Analysis is a process of quantifying the cost estimating uncertainty due to variance in the cost estimating models as well as variance in the technical performance and programmatic input variables
bull Cost Risk Analysis is a process of quantifying the cost impacts of the unfavorable events
Unit III - Module 9 7
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 8
Are Our Costs Realisticbull We claim our costs are realisticbull We claim we estimate costs at the 50th percentile
ndash This should mean we have 50 low and 50 high outcomes
bull But we have an unchanging cost growth pattern showing only 12 of our estimates are high (underrun) and 88 are low (overrun)ndash As shown in the graphic on the next slide
bull The conclusion is inescapable our costs are not realistic
bull This also means our costs are at the 12th percentile not the 50th percentile
What Percentile Are We At Now (And Where Are We Going) R Coleman E Druker P Braxton B Cullis C Kanick SCEA 2009 DoDCAS 2010
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
88 Dollars ( in millions)
Co
st G
row
th F
acto
r
0
1
2
3
4
5
6
7
8
0 5000 10000 15000 20000 25000 30000 35000
lsquoUnit III - Module 9 9
Historical Cost Growth
Average program cost growthRampD 21 Prod 19
Fraction of programs ending on-or-under cost target
7-16
Note This pattern appears to be fractal
Uncohorted dollar weighted
This is the line of no growth
The Cost Growth Factor is 10 meaning (Final Cost)(Initial Cost) = 1
Grew to 75 times planned cost
Risk in Cost Estimating General Introduction amp The BMDO Approach 33rd DoDCAS 2000 R L Coleman J R Summerville M DuBois B Myers
Note apparent bounds
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Sources of Cost Risk
Unit III - Module 9 10
l Security Risksl Critical failure modesl Energy Environmental Risksl Schedule problems and delaysl Inadequate cost estimates
n Process (need to assess contractorrsquos assumptions)
n Models l ldquoNew Ways of Doing Businessrdquol Inflationl Systems Engineeringl Cost Improvement Curve
Assumptions
l State-of-the-Art-Advance (Technology Readiness)
l Technical Risk Sourcesn Physical propertiesn Material Propertiesn Radiation Properties
(emission and reception)l Material Availability Risksl Testing Modeling Risksl Integration Interface Risksl Program Personnell Safety Risksl Software Design Risks
Historical cost data available
Amount of cost risk depends on the Basis of the Estimate
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates of Costbull Funding organizations need best estimate of cost for
bullCostperformance tradeoff studiesbullCostbenefit analysesbullBudget planning
bull But program cost is nebulous heavily impacted bybullTechnological (im)maturitybullProgrammatic considerationsbullSchedule slipsbullUnforeseen events
bull Point cost estimates cannot be ldquocorrectrdquo becausebullEvery cost element contains uncertaintybullTotal system cost is sum of these WBS elements
bull Actual program cost falls within a range surrounding the best estimate (with some degree of confidence)bullThe best we can hope to do is to understand the uncertaintybullUnderstanding the uncertainty will help us make provision for it
Unit III - Module 9 11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 12
Types of Riskbull Cost Growth = Cost Estimating Growth + SkedTech Growth
+ Requirements Growth + Threat Growthbull Cost Risk = Cost Estimating Risk + SkedTech Risk +
Requirements Risk + Threat Riskndash Cost Estimating Risk Risk due to cost estimating errors and the statistical
uncertainty in the estimate ndash ScheduleTechnical Risk Risk due to inability to conquer problems posed
by the intended design in the current CARD or System Specificationsndash Requirements Risk Risk resulting from an as-yet-unseen design shift from
the current CARD or System Specifications arising due to shortfalls in the documents
bull Due to the inability of the intended design to perform the (unchanged) intended mission
bull We didnrsquot understand the solutionndash Threat Risk Risk due to an unrevealed threat eg shift from the current
STAR or threat assessmentbull The problem changedO
ften
impl
icit
or o
mitt
ed
1 2
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates Vs Risk Estimates
Unit III - Module 9 13
DDTampE Flight Unit Production TotalPoint Estimate 63807 5179 6697 70505CER Risk Only - Mean 71176 4805 6426 78018Full Risk - Mean 78937 6490 8091 87430Point Esimtate + 30 82949 6733 8706 91657Full Risk - 70th Percentile 87939 7980 9608 97011
Estimated Total Cost in Millions
Note that the mean is 24 higher than the point estimate and the 70th
percentile is 38 higher than the point estimate
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Cost Risk and Probabilitybull Cost risk or cost uncertainty analysis represents cost as a
probability distributionbull A probability distribution is a mathematical rule associating a
probability a to each possible outcome or event of interestbull There are two ways to present a probability distribution ndash as a
probability density function (ldquoPDF) and as a cumulative distribution function (ldquoS-curverdquo or ldquoCDFrdquo)
Unit III - Module 9 14
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates of Costbull Funding organizations need best estimate of cost for
bullCostperformance tradeoff studiesbullCostbenefit analysesbullBudget planning
bull But program cost is nebulous heavily impacted bybullTechnological (im)maturitybullProgrammatic considerationsbullSchedule slipsbullUnforeseen events
bull Point cost estimates cannot be ldquocorrectrdquo becausebullEvery cost element contains uncertaintybullTotal system cost is sum of these WBS elements
bull Actual program cost falls within a range surrounding the best estimate (with some degree of confidence)bullThe best we can hope to do is to understand the uncertaintybullUnderstanding the uncertainty will help us make provision for it
Unit III - Module 9 11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 12
Types of Riskbull Cost Growth = Cost Estimating Growth + SkedTech Growth
+ Requirements Growth + Threat Growthbull Cost Risk = Cost Estimating Risk + SkedTech Risk +
Requirements Risk + Threat Riskndash Cost Estimating Risk Risk due to cost estimating errors and the statistical
uncertainty in the estimate ndash ScheduleTechnical Risk Risk due to inability to conquer problems posed
by the intended design in the current CARD or System Specificationsndash Requirements Risk Risk resulting from an as-yet-unseen design shift from
the current CARD or System Specifications arising due to shortfalls in the documents
bull Due to the inability of the intended design to perform the (unchanged) intended mission
bull We didnrsquot understand the solutionndash Threat Risk Risk due to an unrevealed threat eg shift from the current
STAR or threat assessmentbull The problem changedO
ften
impl
icit
or o
mitt
ed
1 2
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates Vs Risk Estimates
Unit III - Module 9 13
DDTampE Flight Unit Production TotalPoint Estimate 63807 5179 6697 70505CER Risk Only - Mean 71176 4805 6426 78018Full Risk - Mean 78937 6490 8091 87430Point Esimtate + 30 82949 6733 8706 91657Full Risk - 70th Percentile 87939 7980 9608 97011
Estimated Total Cost in Millions
Note that the mean is 24 higher than the point estimate and the 70th
percentile is 38 higher than the point estimate
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Cost Risk and Probabilitybull Cost risk or cost uncertainty analysis represents cost as a
probability distributionbull A probability distribution is a mathematical rule associating a
probability a to each possible outcome or event of interestbull There are two ways to present a probability distribution ndash as a
probability density function (ldquoPDF) and as a cumulative distribution function (ldquoS-curverdquo or ldquoCDFrdquo)
Unit III - Module 9 14
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 12
Types of Riskbull Cost Growth = Cost Estimating Growth + SkedTech Growth
+ Requirements Growth + Threat Growthbull Cost Risk = Cost Estimating Risk + SkedTech Risk +
Requirements Risk + Threat Riskndash Cost Estimating Risk Risk due to cost estimating errors and the statistical
uncertainty in the estimate ndash ScheduleTechnical Risk Risk due to inability to conquer problems posed
by the intended design in the current CARD or System Specificationsndash Requirements Risk Risk resulting from an as-yet-unseen design shift from
the current CARD or System Specifications arising due to shortfalls in the documents
bull Due to the inability of the intended design to perform the (unchanged) intended mission
bull We didnrsquot understand the solutionndash Threat Risk Risk due to an unrevealed threat eg shift from the current
STAR or threat assessmentbull The problem changedO
ften
impl
icit
or o
mitt
ed
1 2
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates Vs Risk Estimates
Unit III - Module 9 13
DDTampE Flight Unit Production TotalPoint Estimate 63807 5179 6697 70505CER Risk Only - Mean 71176 4805 6426 78018Full Risk - Mean 78937 6490 8091 87430Point Esimtate + 30 82949 6733 8706 91657Full Risk - 70th Percentile 87939 7980 9608 97011
Estimated Total Cost in Millions
Note that the mean is 24 higher than the point estimate and the 70th
percentile is 38 higher than the point estimate
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Cost Risk and Probabilitybull Cost risk or cost uncertainty analysis represents cost as a
probability distributionbull A probability distribution is a mathematical rule associating a
probability a to each possible outcome or event of interestbull There are two ways to present a probability distribution ndash as a
probability density function (ldquoPDF) and as a cumulative distribution function (ldquoS-curverdquo or ldquoCDFrdquo)
Unit III - Module 9 14
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Point Estimates Vs Risk Estimates
Unit III - Module 9 13
DDTampE Flight Unit Production TotalPoint Estimate 63807 5179 6697 70505CER Risk Only - Mean 71176 4805 6426 78018Full Risk - Mean 78937 6490 8091 87430Point Esimtate + 30 82949 6733 8706 91657Full Risk - 70th Percentile 87939 7980 9608 97011
Estimated Total Cost in Millions
Note that the mean is 24 higher than the point estimate and the 70th
percentile is 38 higher than the point estimate
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Cost Risk and Probabilitybull Cost risk or cost uncertainty analysis represents cost as a
probability distributionbull A probability distribution is a mathematical rule associating a
probability a to each possible outcome or event of interestbull There are two ways to present a probability distribution ndash as a
probability density function (ldquoPDF) and as a cumulative distribution function (ldquoS-curverdquo or ldquoCDFrdquo)
Unit III - Module 9 14
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Cost Risk and Probabilitybull Cost risk or cost uncertainty analysis represents cost as a
probability distributionbull A probability distribution is a mathematical rule associating a
probability a to each possible outcome or event of interestbull There are two ways to present a probability distribution ndash as a
probability density function (ldquoPDF) and as a cumulative distribution function (ldquoS-curverdquo or ldquoCDFrdquo)
Unit III - Module 9 14
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 15
Probability Model ndash Distributionbull Normal
ndash Best behavior most iconic
ndash Theoretically (although not practically) allows negative costs which spook some users
ndash Symmetric needs mean shift to reflect propensity for positive growth
bull Lognormalndash A natural result in
non-linear CERsndash Indistinguishable
from Normal at CVs below 25
ndash Skewed
10
4
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 16
bull Triangularndash Most commonndash Easy to use easy to understandndash Modes medians do not addndash Skewed
bull Betandash Rare now but formerly popularndash Solves negative cost and duration issuesndash Many parameters ndash simplifications like PERT
Beta are possiblendash Skewed
bull Bernoullindash Probability is only assigned to two possible
outcomes success and failure (p and 1-p)ndash Simplest of all discrete distributionsndash Mean = pndash Variance = p(1-p)
10
Probability Model ndash Distribution
10
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 17
bull Normalndash If there is a sufficiently large number of independently identically distributed
elements (or not too correlated and not too skewed) then the distribution of total cost is normal due to the Central Limit Theorem
bull Lognormalndash For smaller data sets with large amounts of (positive) correlation and
skewness the lognormal distribution may be a better approximation of total cost
ndash Many total cost distributions in smaller data sets test as both normal and lognormal
bull Triangularndash The sum of triangles is not a triangle so this distribution should generally not
be used to characterize total costbull Convolved
ndash If total cost distribution is derived using simulation then it may not particularly resemble anything In this case we simply have a convolved or simulated total cost distribution
Probability ndash Top-Level Distribution
NEW
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
S-Curves and Confidence Levels
Unit III - Module 9 1818
100
70
25
Probability(Confidence
Level)
Cost Estimate(Dollars)
50
The S curve is the cumulative probability distribution coming out of the statistical summing process
70 confidence that project will cost indicated amount or less
Provides information on potential cost as a result of identified project risks
Provides insight into establishing reserve levels
X Y
MR
X = Point estimateZ = Project requirementY = Cost estimate where there is a 70 change that
final actual cost will be less than cost estimateY-Z= Manage reserve (MR) which is REQUIRED to meet 70 CL
Z
X not always = Z but is based on same content
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
19
Example ndash Normal Distribution
bull Normal distribution with mean equal to 100 and standard deviation equal to 20
bull Median and mode also equal 100bull The 80th Percentile is 117
40 60 80 100 120 140 160
80 of distribution(area under curve)is less than 117
117
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
20
Confidence Levelsbull The output of a cost risk analysis is a probability
distributionndash Typically displayed as a cumulative distribution function or
ldquoS-curverdquondash Confidence levels are simply percentiles of the distribution
0
10
20
30
40
50
60
70
80
90
100
30 60 90 120 150 180
Conf
iden
ce L
evel
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Measuring Parametric Model Uncertainty
bull When developing estimates using cost-estimating relationships (CERs) there are two major sources of uncertaintyndash Technical (model input) uncertaintyndash Estimating uncertainty
bull Technical uncertainty reflects the uncertainty inherent in the model inputsndash Typical parametric model inputs include weight lines of code
heritage and other performance and management inputs
bull Estimating uncertainty measures the uncertainty inherent in the model due to the modelrsquos inability to capture all aspects of variability in the data
21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Parametric Model Examplebull The NASAAir Force Cost Model (NAFCOM) is a
parametric model for estimating acquisition costs for spacecraft and launch vehicles
bull The CERs in NAFCOM have the formCost = C WeightVNewDesignWTechnicalXManagementY ClassZ
bull There is technical uncertainty about each of the parametersndash Parametric models are applied early in a program so there
is significant amount of uncertainty in the weightndash Heritage is also tricky it is inherently subjective and project
personnel are typically optimisticndash Technical characteristics can also change as a program
matures and requirements change
22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Weight Uncertaintybull Aerospace weight growth study average satellite
weight growth from program inception to launch is 28
23
Aerospace Weight Growth Data
-10
0
10
20
30
40
50
60
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60 64 68 72 76 80 84 88 92 96 100
Percent Program Completion
Perc
ent W
eigh
t Gro
wth
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertaintybull CERs do not perfectly fit the historical data
ndash There are non-repeatable random effects that cannot be predicted
bull For examplendash Parts breaking during testingndash Strikes which are difficult to predict
bull This results in an underlying uncertainty distribution about an estimatendash The outcome of a CER represents only one point on an
uncertainty distributionbull Typically mean or median depending upon the CER
methodology
24
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (2)
25
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Estimating Uncertainty (3)bull In the case of a linear regression the estimating
uncertainty follows a normal distributionndash The estimate is the mean and the standard error of the
estimate from the regression is the standard deviation of the normal distribution
bull In the case of a log-transformed linear regression the estimating uncertainty follows a lognormal distributionndash The estimate is the median of the lognormal and the
standard error of the estimate from the regression is the standard deviation of the normal distribution (in log-space)
bull There is additional uncertainty about an estimate based upon the distance of what is being estimated from the mean of the input datandash This results in a prediction interval
26
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull Risk is typically applied at the same level at
which we estimate cost that is at the Work Breakdown Structure (WBS) level
bull The estimates for each WBS are developedndash These estimates sum to yield the total system cost
bull For an n-element WBS the total cost is the sum of the cost for each of the n elements
27
nXXXCostSystem +++= 21
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
WBS Examplebull Launch Vehicle
ndash Structurebull Vehicle Structurebull Tank Structure
ndash Thermal Controlbull Active Thermal Controlbull Induced Thermal Controlbull Tank Thermal Control
ndash Main Propulsion Systemndash Liquid Rocket Enginendash Electric Power and Distributionndash Command Control and Data Handlingndash System Integration
28
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
WBS and Risk
29
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Aggregating Riskbull This leads us to the issue of aggregating the risks across the
WBSbull For example if we want to determine the 70th percentile of the
overall cost risk distribution how do we proceed bull Note that we cannot simply add the 70th percentiles of each of
the cost risk distributionsbull We can add means and variances (taking correlation into
account more on that later) but that is all
Variance of Total Cost =
30
Mean of Total Cost = ( )E X E Xk
k
n
kk
n
kk
n
= = =aringaelig
egraveccediloumloslashdivide
= =aring aring1 1 1
m
Var X kk
n
=aringaelig
egraveccediloumloslashdivide1
= s r s skk
n
iji
j
i jj
n2
1 1
1
22+aring aringaring
= =
-
=
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Addbull As a notional example consider two independent normal
distributionsndash One distribution has mean equal to 100 and standard deviation
equal to 20ndash The second distribution has mean 300 and standard deviation of 80
bull The sum of two independent normally distributed random variables is also normally distributed
bull To combine the two distributions we add the means and add the variancesndash Total mean is
ndash Total standard deviation is
31
400300100 =+
8258020 22 raquo+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (2)bull The 80th percentiles for the individual distributions are
117 and 367 respbull The 80th percentile of the combined distribution is
469 but the sum of the two 80th percentiles is 484bull To see why this is the case note that the percentiles
of a normal distribution are determined by the mean and the standard deviationndash The standard deviations are not added when normal distributions
are combined rather the variances are combinedndash Sum of variances is ndash The standard deviation of this sum is
32
22 ba +22 ba +
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Percentiles Do Not Add (3)bull Since bull We can write
bull The left side of this inequality represents the risk of the combined distributions while the right side represents the sums of the individual risks
bull Combining two missions that are independent results in a diversification of riskndash The total portfolio is not as risky on a relative basis as each
individual project
33
2222 2 bababa ++lt+
baba +lt+ 22
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Methods for Aggregating Riskbull Since percentiles do not add we have to find other
ways to aggregate riskbull One method is to use Monte Carlo simulationbull Unless cost risk is represented by normal
distributions for all WBS elements summing the means and variances does not result in a completely accurate depiction of overall system cost risk
bull The normal distribution is not a realistic distribution for representing risk in many or even most situations since regarding project risk more can go wrong than can go right which means the risk is NOT symmetric unlike the Gaussian bell curve
34
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 35
Monte Carlo Simulationfor Risk Analysis
bull When risk analysis is performed multiple risks are gatheredndash These risks may have various probability
distributions
bull Monte Carlo is the most commonly accepted way to produce an accurate characterization of the distribution of the combined effect of these many risksndash Using this distribution we can produce percentiles
for cost impacts due to risk occurrences
10
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo Simulationbull Simulation is a means for solving problems that are
analytically intractablendash Involves repeated random sampling
bull It was developed during the Manhattan Project as a way to model the interactions inside an atom during nuclear fissionndash Allowed for more sophisticated mathematical modeling of atomic
phenomena
bull Most computer packages use pseudo-random number generators that produce seemingly random results from an initial seedndash Example is the linear congruential generator
36
( ) mbaXX nn mod1 +=+
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Using Random Numbers to Simulate Distributions
bull WBS cost risk is typically measured by a probability distributionbull The most common method for simulating a probability
distribution is the inverse methodbull This involve obtaining a simulated random value from a uniform
distribution and then inverting this through the cumulative distribution function (CDF) to obtain the simulated value
bull As an example for a uniform distribution on the interval [0100]
bull In this case the CDF random value of 05 translates to 10005 = 50
37
( )100
1=xf ( )
100xxF = ( ) xxF 1001 =-
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution
bull For a triangular distribution with low = 0 most likely = 50 and high = 150 the CDF is given by
bull The inverse function is given by
38
( )iumliumlicirc
iumliumliacute
igrave
poundlt--
poundpound=
1505021
1500050
50015000
2
2
2
xxx
xxxF
( )iumliumlicirc
iumliumliacute
igrave
poundltdivideoslashouml
ccedilegraveaelig +--
poundpound=-
131
21
37501
250017500150
3107500
1
xx
xxxF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Simulating a Triangular Distribution (2)
bull If the simulated uniform value is equal to 02 the corresponding inverse value from the CDF inverse is approximately 387
39
0
01
02
03
04
05
06
07
08
09
1
0 30 60 90 120 150
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Monte Carlo and the WBSbull Simulation involves repeated trials
40
WBS Draw1 2 3 4 5
System 2912 3566 4390 2954 4655 Sum
Structure 80 105 55 80 57 SumVehicle Structure 50 70 30 40 20Tank Structure 30 35 25 40 37
Thermal Control 32 36 35 29 33 SumActive Thermal Control 10 15 12 7 10Induced Thermal Control 12 14 13 14 14Tank Thermal Control 10 7 10 8 9
Main Propulsion System 100 150 125 140 100
Liquid Rocket Engine 1000 1100 1800 900 2200
Electric Power and Distribution 100 125 150 75 90
Command Control and Data Handling 200 250 225 230 275
System Integration 1400 1800 2000 1500 1900
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trialsbull Monte Carlo involves a repeated number of trialsbull The greater the number of trials the more accurate the
simulation will approximate the overall cost risk behavior of the system
bull After 100 trials the histogram looks very jagged
41
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (2)
42
1000 Trials
10000 Trials
50000 Trials Additional trials give a more accurate depiction of the final result
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (3)bull If we want to simulate the underlying
mean within 1 with 95 confidence for example then we require
43
divideoslash
oumlccedilegrave
aelig poundpound-
=
dividedividedivide
oslash
ouml
ccedilccedilccedil
egrave
aeligpound
-pound
-=
divideoslashouml
ccedilegraveaelig poundpound=
nZ
n
nnX
n
X
n
n
011
010Pr
011
010Pr
011990Pr950
___
___
sm
sm
sm
sm
sm
mm
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (4)bull This means that the number of trials required is determined by
bull Solving for n we find that at least
trials are neededbull Note that this term involves the coefficient of variation of the
distribution which means that distributions with less dispersion require fewer trials to accurate simulate
44
96101
=ns
m
ms41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (5)bull Simulating percentiles requires more trials than for simulating
meansbull Let Pn represent the number of trials in the sample at or below
the 70th percentile of the underlying distribution (suppose this is known)
bull By the Central Limit Theorem Pnn follows a normal distribution with mean equal to the 70th percentile and variance equal to
bull By a similar process for the mean we find that the number of trials needed to simulate the 70th percentile within 1 with 95 confidence is
45
( )n
FF 700700 1-
n
n
PPn -41638
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Number of Trials (6)bull If Pn 070 for all n then n is approximately
bull For the sake of comparison suppose CV = 03 (reasonable approximation for hardware development) then the number of trials needed to simulate the mean with the same accuracy is then ldquoonlyrdquo 11525
bull More trials are needed to accurately simulate percentiles than means
46
( ) 46416703041638
70704168341638 =divide
oslashouml
ccedilegraveaelig=divide
oslashouml
ccedilegraveaelig -
=-
nnn
PPn
n
n
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Methods of Simulationbull Monte Carlo simulation is a standard term for
simulationbull However there is an improvement on Monte Carlo
that improves the accuracy of the results of a simulation for a given number of trials
bull This method is call the Latin Hypercube approach to simulationndash Similar to Monte Carlo but an equal number of draws are taken
from a set of subintervals for example dividing the interval (01) into [00001) [0102)hellip [0910]
ndash Instead of 10 trials from [01] we have one trial from each subinterval
ndash Rough rule of thumb is Latin Hypercube takes 30 fewer trials to achieve similar accuracy to Monte Carlo
47
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Probability Model ndash Correlationbull Correlation exists between work breakdown structure
(WBS) cost element costs and between cost and schedule
bull Correlation is a necessary consideration in cost risk analysis
bull Correlation is a number that can vary between -1 and +1 and represents the strength of the relationship between the two variablesndash If both variable have a tendency to move in the same direction
correlation is positivendash If when one variable increases the other tends to decrease and
vice-versa the correlation between the two variables is negative
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Correlation And Causationbull Note that correlation is not causation however
causation implies correlationndash For example shark attacks and ice cream sales at the beach
bull For example for a launch vehicle structure increasing the diameter will not only result in an increase in the structures cost but will also result in higher costs for thermal coating (paint etc)
bull Also simply because two subsystems do not appear to be related does not mean they are not correlatedndash For example structures and computer equipment seemingly
have little relation to one another however an external factor such as a funding cut or a strike will impact all subsystems leading to delays and increasing cost across the board
ndash Thus everything in a WBS has some correlationUnit III - Module 9 49
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlationbull Percent that total cost standard deviation is underestimated
when correlation assumed to be 0 instead of r given nWBS elements
Unit III - Module 9 5050
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
0
20
40
60
80
100
0 01 02 03 04 05 06 07 08 09 1Actual Correlation
Perce
nt Un
deres
timate
d
n = 10
n = 30
n = 100n = 1000
Reference 32nd Annual DOD Cost Analysis Symposium Advanced Training Session ldquoWhy Correlation Matters in Cost Estimatingrdquo Stephen A Book
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
The Importance of Correlation (2)bull For example if correlation is assumed to be 0 for all
elements in a 30 element WBS and the actual correlation is 20 then the total cost standard deviation is underestimated by 60
bull For example assume the total cost is modeled as a normal distribution with mean = 100
bull If correlation is assumed to be zero with total cost standard deviation = 12 when correlation is actually 20 then the true underlying standard deviation is actually 30
bull 70th percentile with zero correlation assumption is 106
bull 70th percentile with 20 correlation is 116 Unit III - Module 9 51
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Measures of Correlationbull Two measures of correlation are commonly used in
cost risk analysesndash Pearsonrsquos product-moment correlation coefficient ndash Spearmanrsquos rank correlation coefficient
bull Garvey and others recommend using Pearsonrsquos product-moment correlation coefficient correlation for cost risk analysesndash Method of moments uses this but there are two options in
simulationndash Popular simulation packages like Risk and Crystal Ball use
rank correlation but in practice this has been found to make little difference
ndash Pearson product-moment correlation has been incorporated into Risk+
52
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Pearson Correlationbull Pearsonrsquos product-moment correlation measures
degree of linearity between two random variables bull If two random variables are perfectly linear (eg Y =
aX + b) with positive slope then the Pearson product-moment correlation between X and Y denoted by rXY is +1
53
X X
Y Y Y
XNegative Correlation Positive Correlation No Correlation
middotmiddot
middotmiddot
middotmiddot middotmiddot
middotmiddot
middot
middot
middotmiddotmiddot
middotmiddot
middotmiddot
middotmiddot middot
middot
middot
middotmiddotmiddotmiddotmiddotmiddotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddot
middotmiddot middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddotmiddot
middotmiddotmiddot
middotmiddotmiddot
middotmiddot
middot
middot middotmiddot
middot middotmiddotmiddotmiddot middotmiddot middot middotmiddot middot middot
middotmiddot middot
middotmiddotmiddotmiddotmiddot middot middotmiddot
middotmiddot
middot
middotmiddot
middotmiddot middot
middot middot middot
middotmiddot middot
middotmiddotmiddot
middotmiddot
middot middot
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Spearman Correlationbull Spearmanrsquos rank correlation measures the
monotonicity between two random variablesbull If two random variables are perfectly monotonically
increasing (eg Y = X2) then the Spearman rank correlation denoted by rXY is +1
bull The two measures can be significantly different
54
10 20 30 40 50 60
100
200
300
096 097 098 099 1
1
060
020
040
080
Eff
ort (
Staf
f-M
onth
s)
Software Size I KDSI
y = 28z12y
z x
yy = x100
0 pound x pound 1
Figure 5-9A Figure 5-9B
Pearson Correlation = 09976 Rank Correlation = 1
Pearson Correlation = 024 Rank Correlation = 1
X(x ) ~ Unif(01)
Reference Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective Paul R Garvey
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 55
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
200
250
300
350
400
1000 1200 1400 1600 1800 2000
Recurring Production
SEPM
Functional Correlationbull Old No Functional
Correlation Simulation run with WBS items entered as values
bull New Simulation run with functional dependencies entered as they are in cost model
CorrelatedNot Correlated
Note shift of mean and increased variability
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Risk Measuresbull ldquoWestern Europe conquered the word because of a
technological revolution that started because of attempts to measure the worldrdquo ndash Phillipe Jorion ndash In the same way attempts to measure risk will lead to better project
managementbull Risk management for DoD and NASA agencies focuses
primarily on funding projects to a single percentile ndash NASA policy explicitly mentions 70 and 50ndash This is known as ldquoValue at Riskrdquo and is prevalent in the banking and
financial industry
56
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Value at Risk (VaR)bull VaR is a percentile of the cost risk
distributionndash VaR funding at the 70th percentile means that
there is a 30 chance of final project cost exceeding the funded amount (budget)
bull VaR has some meritsndash Common measure can be used to compare
different projects and programsndash Easily understood by senior managers and
decision makers
57
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Why 70th or 80th Percentilebull Typical policy guidance involves funding at the 70th or 80th
percentile Why fund as such a low percentilebull There is some incentive to not over-fundbull For a normal distribution an inflection point occurs near the 80th
percentile meaning that for values above 80 confidence achieving higher levels of confidence can only be achieved by spending greater dollar amounts at an increasing rate
58
-05-04-03-02-01
00102030405
0 05 1 15 2 25 3
Standard Deviations from Mean
Normal Distribution
First DerivativeSlopeVelocity
Second DerivativeConvexityAcceleration
Inflection point at one standard deviation above the mean (~84th
percentile)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Funding Above Inflection Pointsbull As the percentile goes above the 80th percentile the amount
needed to achieve that higher level of confidence begins to increase at an increasing rate as is evident from the graph
59
0
10
20
30
40
50
60
70
50 55 60 65 70 75 80 85 90 95 100
$ Ab
ove
MEa
n to
Ach
ieve
Per
cent
ile
Percentile (Confidence Level)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Why Budget to the 70th or 80thbull Government projects are funded at the 70th or 80th percentilesbull Why not the 90th or 99th percentile Wonrsquot that help prevent
overrunsndash As we have seen percentiles do not add across programs
bull The sum of the 80th percentiles does NOT equal the total 80th percentile
ndash This is because the individual programs are riskier than the entire suite of missions
bull Known as the ldquoPortfolio effectrdquo (more or this later)
ndash Funding at the 80th gives confidence that the entire portfolio will be adequately funded
bull And since there is an inflection point near the 80th percentile funding above that level quickly becomes cost prohibitive
60
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Inflection Points for Lognormalbull For a lognormal the inflection point
variesndash Coefficient of variation equal to 10 equates to an
inflection point at the 70th percentile
61
0
10
20
30
40
50
60
70
050 060 070 080 090 100
Coef
ficen
t of V
aria
tion
Percentile of the Inflection Point
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Issues with Percentile Fundingbull However risk management does not stop at a
specific percentilendash What happens when the 70th percentile is exceeded
bull Any effective risk management policy should prescribe what happens after a bad event occursndash With VaR we only know when a bad event has occurred ndash Also funding at a relatively low level means we have no true
sense of the real risks that should be guarded against
62
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
ldquoHere There Be Dragonsrdquobull On old maps uncharted territory was often
marked with monsters or dragons
63
A section of the Carta Marina by Olaus Magnus 16th Century
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Distribution Comparisonbull Four different distributions with the same 70th
percentile
bull Should we expect the tails to be similar
64
0
10
20
30
40
50
60
70
80
0 100 200 300 400 500 600
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison Tail Behavior
bull The normal and triangular distributions have similar tails but the lognormal and Pareto have much fatter right tails
65
70
75
80
85
90
95
100
500 1500 2500 3500 4500 5500
Perc
entil
eCo
nfid
ence
Lev
el
Triangular
Normal
Lognormal
Pareto
Here There Be Dragons
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Distribution Comparison ndash Tail Behavior (Table)
bull There is significant tail risk for the lognormal and the Paretobull The 99th percentile for the Pareto is 1000 times greater than
the 70th percentilendash Risk seen in financial markets and in catastrophic risks like natural
disasters bull Four different projects follow the four different risk profiles seen
in this example should clearly have different risk management approaches but 70th percentile funding indicates the same risk management approach for all four
66
Percentile Triangular Normal Lognormal Pareto 70th $600 $600 $600 $600 80th $680 $660 $770 $1430 90th $770 $750 $1070 $5880 95th $840 $820 $1410 $23750 99th $930 $950 $2350 $600000
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Percentile Budgeting is Not Risk Management
bull Percentile budgeting ne effective risk management policybull Suppose that you are shopping for a new car You
mention that safety is your top concern The salesman says he has a great safe car available You ask about the air bags Do they work The salesman answers ldquoOf course Seventy percent of the time they work fine Only 30 of the time the air bags will fail to deployrdquo Would you buy such a car Hedge fund manager David Einhorn has stated ldquoRisk management is the air bag that must always work but only in the multi-sigma event where you have an accidentrdquo
67
Risk management is the air bag that must always work but only in the multi-sigma event where you
have an accident
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Level Analysisbull Schedules are risky toobull For 98 recent NASA missions average
schedule growth was 38ndash Not as high as cost growth but still significant
68
0
5
10
15
20
25
30
35
40
Schedule
Cost
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Schedules Impact Costbull Changes in schedule can have enormous
impacts on cost ndash Schedule slips can lead to cost increases
bull Fixed programmatic costs must be paid on a regular basisndash Achieving schedule milestones early can result in cost
savings
bull Schedule is an important consideration in cost analysis including cost risk analysis
bull However we must be careful in correctly integrating cost and schedule riskndash Schedule is not simply an input but is dependent on many of
the same factors that drive cost
69
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
The Relationship of Cost to Schedulebull Cost and schedule are positively correlated
ndash A larger program with bigger scope with take longer to complete than a smaller simpler program
bull However longer schedules do not always equate to higher cost nor shorter schedules with lower costndash Highly compressed schedules relative to the
optimal schedule can lead to large overrunsbull Compressed schedules are commonplace for
many planetary missions
70
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12The Impact of Schedule GrowthCompression on Cost
bull Compressed schedules lead to cost growth and based on existing research is actually more expensive than schedule slippages
71
0
20
40
60
80
100
120
140
160
-50 -25 0 25 50 75 100
Percentage Change in Schedule
Perc
enta
ge C
hang
e in
Cos
t due
to
Sche
dule
Cha
nge
Source Smart CB ldquoCost and Schedule Interrelationshipsrdquo 2007 NASA Cost Symposium
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Schedule as an Independent Variable
bull Why not simply add schedule as an input to cost-estimating relationships (CERs)
bull Schedule is often treated as an independent variable ndash schedules are often determined independently of cost considerationsndash Cost may not be considered explicitly although budgets and
schedules are determined based on scope and other common factors
bull In 2006 SAIC undertook a research task to determine the impact of schedule on cost and added schedule as an independent variable to the CERs in the NASAAir Force Cost Model (NAFCOM) to see if schedule provided any improvement in goodness of fit
72
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Schedule is a Dependent Variablebull Since NAFCOM includes an exhaustive list of cost drivers the inclusion
of schedule should give an indication of whether there is any additional independent value added from schedule independent of the cost drivers
bull In the study schedule was added to 10 subsystem-level CERs
ndash In each case the addition of schedule did NOTimprove the goodness of fit and schedule was NOT found to be a statistically significant cost driver (given the presence of the other variables) for any of the 10 CERs
bull Standard errors for the CERs increased due to decrease in degrees of freedom
bull Thus schedule as an independent variable did not provide any additional explanation of variation in historical cost
bull This is because schedule is truly NOT an independent variable but a dependent variable and is a function of the same factors that drive cost
73
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Incorporation of Schedule in Cost Risk
bull When using subjective methods care must be taken in applying schedule to cost risk analysis
bull Since schedule is a dependent variable treating it independently from other factors when assessing its impact on cost can lead to a double count of independent variables in the cost risk analysis
bull Direct inclusion of schedule in a cost risk analysis effectively doubles the influence of each cost driver
ndash Incorrectly leads to broader risk ranges
74
CostWeight
HeritageNewness
Technology Readiness Schedule
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Subjective Cost Risk Methods and Overinclusion of Risk Factors
bull Subjective methods are suitable when data are scarce or not availablebull However care must be taken not to treat correlated sources of risk
such as schedule as independent of each otherbull Example ndash a subjective risk analysis incorporated technology readiness
design risk requirements stability integration risk and schedule and treated each source of risk as independent
bull Risk triangle high value without schedule risk was 31 times the point estimate
bull Risk triangle high value with schedule risk added on top of other sources of risk treated independently was 34 times the point estimate
bull Risk triangle taking correlation between sources of risk into account in a rigorous statistical manner has a high value 15 times the point estimate less than half the effect when treating sources of risk as independent
75
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Fundamental Equation of Cost and Schedule Interrelationships
bull The Fundamental Equation of the Relationship Between Cost and Schedule is
Time = Moneybull This is a well known adage that time is money Your boss at
work may even mention this to you from time to time or if you are the boss you are all too aware of this iron law
bull Because time is money cost and schedule risk should be modeled jointly not with schedule as an independent cost driver
bull Note that cost and schedule are not perfectly correlatedndash Cost can sometimes ldquobuy downrdquo schedule extra money can be
spent to meet a deadline since in some cases additional effort to hasten the completion of a task
76
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12How to (Correctly) Incorporate Schedule into Cost Risk Analysis
bull The best way to model cost and schedule risk is to develop a joint cost and schedule model based on the same data
bull Since such models are not currently available the next best approach is to model cost and schedule independently and then integrate cost and schedule risk analyses using joint probability distributions taking into account correlation between cost and schedulendash This approach pioneered by Paul Garvey and Steve Book is
considered industry best practicendash This is a practical method mathematically tractable and follows
naturally from the Fundamental Equation of Cost and Schedule Interrelationships
77
Book S A ldquoQuantifying the Relationship Between Cost and Schedulerdquo The Measurable News Winter 2007-2008 pp 11-15Garvey PR Probability Methods for Cost Uncertainty Analysis A Systems Engineering Perspective New York Marcel Dekker 2000 pp 308-335
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Joint Lognormal and Normal Distributions
bull Cost and schedule are typically modeled as some combination of Normal and Lognormal probability distributionsndash Studies by The Aerospace Corporation the MITRE
Corporation Wyle Labs and SAIC have found that the Lognormal probability distribution almost always serves as a good model for both the schedule distribution and the cost distribution
ndash The Normal distribution may also be appropriate in some instances and depending upon the level of confidence at which budgets are set may even be more conservative and logically consistent than the Normal distribution
ndash Joint Lognormal and Normal distributions are defined by a pair of mean and standard deviations and a level of correlation between the two
78
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Modeling Cost and Schedule as a Bivariate Distribution
bull In the approach set forth by Book and Garvey cost risk is assessed and the top-level distribution is modeled as a Lognormal (or Normal) distribution schedule risk is assessed and the overall schedule risk is modeled as a Lognormal (or Normal) distribution
bull The two distributions are combined into a Bivariate distribution by assigning a correlation between the two and by treating the two distributions as the marginal distributions of a Bivariate Lognormal distributionndash Other combinations include a Bivariate Normal or a Bivariate
Normal-Lognormal
79
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal Distribution for Cost and Schedule
bull Cost is often observed to be Lognormal when strong positive correlations are present among the systemrsquos WBS cost element costs
bull A systemrsquos schedule is often observed (from Monte Carlo simulations) to be Lognormal if it is the sum of many positively correlated schedule activities in an overall schedule network
bull If Lognormal distributions characterize a systemrsquos cost and schedule then the Bivariate Lognormal could serve as an assumed model of their joint distributionndash In this case the marginal distribution of each of cost and schedule is
Lognormalbull Other possibilities for modeling cost and schedule as a Bivariate
distribution include a Bivariate Normal or a Bivariate Normal-Lognormal distributionndash All these combinations are analytically tractable
80
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Bivariate Lognormal for Cost and Schedule
bull Contour Plots of a Bivariate Lognormal
81
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule
bull Cost and schedule are positively correlated but not perfectly sondash For example it is possible to have a schedule underrun and a cost
overrunbull Based on cost and schedule growth data for 50 completed NASA
programs the correlation between development cost and development schedule is 70
82
00
200
400
600
800
1000
1200
-500 00 500 1000 1500 2000 2500Historical Cost Growth
Hist
oric
al S
ched
ule
Gro
wth
r = 715
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Correlation Between Cost and Schedule Caveats
bull Based on historical data recommended correlation between cost and schedule is 70 but this could be higher or lower depending upon program circumstancesndash If program is budget constrained cost and schedule will be
more highly correlated (go as you pay)ndash If program is schedule constrained cost and schedule will
have lower correlation perhaps even negative correlation ndash Smaller programs have smaller correlation between cost and
schedule riskbull Relatively easy to ldquohurry uprdquo and finish on time or even early if
needed
83
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Conditional Confidence Levelsbull Given a programrsquos project schedule we can determine the cost
risk distribution conditional upon the given schedulebull Conditional cost S-curves are steeper than unconditional
(marginal) cost S-curves between some uncertainty has been removed because schedule is fixed
84
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Joint Confidence Levelsbull Joint 65 confidence represents the probability of
achieving both 65 confidence in cost AND 65 confidence in schedulendash This is different than achieving 65 cost confidence alone
or achieving 65 schedule confidence alonebull Due to the imperfect correlation between cost and schedule it
is possible to achieve a 65 cost confidence with a lower level of schedule confidence
bull Restricting the schedule confidence thus requires additional $ to achieve a given level of confidence
85
X
Y
Possible Values for Which X gt a Possible Values for
Which X gt a and Y gt b
a
b
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Joint Confidence Levels Versus Cost Confidence Levels Only
bull Example of the differences between cost confidence schedule confidence conditional cost confidence and joint cost and schedule confidence
86
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
50 Confidence
70 Confidence
65 Cost Confidence AggressiveSchedule65 Cost ConfidenceScheduleGiven 65 Schedule ConfidenceUnconditional 65 CostConfidenceUnconditional 65 ScheduleConfidence
65 Cost Confidence 65 Joint Cost and Schedule
Confidence for 65 Schedule Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Joint Cost and Schedule Confidence
bull Can examine pairs of cost and schedule that achieve a given level of confidence
bull Provides insights into the trade-offs between cost and schedule for a given level of confidencendash Possible to increase cost confidence by giving a
project more time in some instances or possible to ldquobuy downrdquo schedule
bull But these have limits and are shown in the iso-cost and schedule confidence curves
87
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Cost and Schedule Trade-offs for a Given Level of Joint Confidence
bull Hypothetical Example
88
Cost (RY$M)
Schedule (Months)
$100 48$99 48$89 49$87 50$85 51$85 52$84 53$83 55$83 55$83 56
Cost and Schedule 65 Confidence Iso-Curve
4748495051525354555657
$80 $82 $84 $86 $88 $90 $92 $94 $96 $98 $100
Cost (RY$ M)
Sche
dule
(Mon
ths)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Policy Considerations for Confidence Levels
bull For large programs the difference between the amount of money needed to achieve 65 cost confidence independent of schedule and joint 65 cost and schedule confidence can represent hundreds of millions and even billions of dollars
89
Joint Cost-Schedule Confidence Iso-Curves
Cost (RY$M)
Sche
dule
(Mon
ths)
65 Confidence
65 Cost Confidence AggressiveSchedule
65 Cost ConfidenceScheduleGiven 65 Schedule Confidence
Unconditional 65 CostConfidence
Joint Cost and Schedule Confidence for 65 Cost Confidence and Program Baseline Schedule is Less Than 35
65 Cost Confidence Independent of Schedule Cost Needed to Achieve
Both 65 Joint Cost and Schedule Confidence for the Schedule with 65 Confidence
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Copulasbull More generally any combination of distributions can
be used (more than just normal and lognormal)ndash Termed ldquocopulardquo from the Italian word for ldquocouplerdquondash Copulas are a way to combine marginal distributions of any
number and type into a joint distributionbull More formally a copula is a multivariate joint distribution defined
on the n-dimensional unit cube [0 1]n such that every marginal distribution is uniform on the interval [0 1]
bull (The bivariate form of) Sklarrsquos theorem states that for any bivariate distribution function H(x y) let F(x) = H(x infin) and G(y) = H(infin y) be the univariate marginal probability distribution functions Then there exists a copula C such that H(xy)=C(F(x)G(y))
ndash Copulas are widely used in risk management for investments and insurance
bull Considered industry best practice (Li 2000)bull Marketed as part of JP Morganrsquos CreditMetrics suite of tools
(Gupton et Al 1997)90
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis
bull A parametric model based on a consistent data set with a set of overlapping cost and schedule drivers is a cleaner way to model cost and schedule
91
CostWeight
HeritageNewness
Technology Readiness
Schedule
0
20
40
60
80
100
Confidence
Joint Cost and Schedule Confidence
Schedule
Cost
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Lognormal Distribution PDF and CDF
0102030405060708090100
200 300 400 500 600 700 800 900 1000Cost $
Con
fiden
ce L
evel
0
00002
00004
00006
00008
0001
00012
00014
00016
Prob
abili
ty D
ensi
ty
CDFPDF
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Parametric Joint Cost and Schedule Analysis (2)
bull Joint cost and schedule risk analysis produced naturally by the integrated modelndash Addresses several issues with the copula
approachbull Tail dependencybull Correlation ndash no need to assign correlation
between cost and schedulebull Consistency of cost and schedule models
92
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Involves assigning cost and schedule risks to a resource-loaded schedule
networkbull Bottom-up approach
ndash More labor intensive than parametrics but can also provide more insight
ndash Can answer questions than parametric typically cannot such as how much do reserve requirements decrease if a specific risk is retired
bull Currently being implemented in Constellation programbull Has same issue as all bottom-up models hard to count all the risks and model
them discretely
ndash As a result there is a tendency to reflect less risk than indicated by history
ndash ldquoThere are more things in heaven and earth Horatio than are dreamt of in your philosophyrdquo William Shakespeare Hamlet Act I Scene V
93
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12Resource-Loaded Schedules Vs Parametrics
bull Parametric most appropriate for early estimatesndash But also good for high-level checks as project progresses
bull As plans take shape lower level modeling may be more appropriate
94 1
CONCEPTUALDEVELOPMENT
PROJECTDEFINITION DESIGN DEVELOPMENT OPERATIONS
PARAMETRIC PARAMETRIC
ANALOGY
ENGINEERING (Bottoms Up)
$PARAMETRIC
DETAILED
$
Pre Phase A Phase B Phase CDPhase A Phase EPre Phase A Phase B Phase CDPhase A Phase E
Risk = f(Cost Estimating Relationship Inputs eg mass power data rate TRL new design etc)
Risk = f(cost and schedule variation of the analogous elements)
Risk = f(variation of detailed inputs eg labor hours rates materials etc)
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Resource-Loaded Schedulesbull Methodology has been demonstrated in other
industriesndash Construction oil and gasndash Analysis has not been traditionally done on highly uncertain
complex development projects bull Methodology focuses on project baseline schedule
ndash In baseline schedule all tasks are resource loaded to represent a baseline schedule and cost standpoint
ndash Resource loaded schedule is ldquorolledrdquo up to a manageable level
ndash Based on the risk and uncertainties identified schedule distributions are applied to each rolled up task
ndash Cost uncertainties are also applied to rolled up tasksbull Uncertainties include labor rate man power etc
ndash A Monte Carlo simulation is run to develop joint cost-schedule distributions
95
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-
copy 2002-2013 ICEAA All rights reserved
v12
Unit III - Module 9 96
Execution ndash Cross Checksbull Means The mean cost growth factor for WBS items
can be compared to history as a way to cross check results
bull CVs The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results
bull Inputs Checks are performed on inputs or other parameters to see if historical values are in line with program assumptionsndash Example Historical risk scores can be compared to program
risk scores to see if risk assessors are being realistic and to see if the underlying database is representative of the program
11
Presented at the 2017 ICEAA Professional Development amp Training Workshop wwwiceaaonlinecomportland2017
- Cost and Schedule Risk Analysis Basic
- Acknowledgments
- Unit Index
- Risk Overview
- Agenda
- Introduction
- Risk Analysis Terms
- Are Our Costs Realistic
- Historical Cost Growth
- Sources of Cost Risk
- Point Estimates of Cost
- Types of Risk
- Point Estimates Vs Risk Estimates
- Cost Risk and Probability
- Probability Model ndash Distribution
- Probability Model ndash Distribution
- Probability ndash Top-Level Distribution
- S-Curves and Confidence Levels
- Example ndash Normal Distribution
- Confidence Levels
- Measuring Parametric Model Uncertainty
- Parametric Model Example
- Weight Uncertainty
- Estimating Uncertainty
- Estimating Uncertainty (2)
- Estimating Uncertainty (3)
- Aggregating Risk
- WBS Example
- WBS and Risk
- Aggregating Risk
- Percentiles Do Not Add
- Percentiles Do Not Add (2)
- Percentiles Do Not Add (3)
- Methods for Aggregating Risk
- Monte Carlo Simulationfor Risk Analysis
- Monte Carlo Simulation
- Using Random Numbers to Simulate Distributions
- Simulating a Triangular Distribution
- Simulating a Triangular Distribution (2)
- Monte Carlo and the WBS
- Number of Trials
- Number of Trials (2)
- Number of Trials (3)
- Number of Trials (4)
- Number of Trials (5)
- Number of Trials (6)
- Methods of Simulation
- Probability Model ndash Correlation
- Correlation And Causation
- The Importance of Correlation
- The Importance of Correlation (2)
- Measures of Correlation
- Pearson Correlation
- Spearman Correlation
- Functional Correlation
- Risk Measures
- Value at Risk (VaR)
- Why 70th or 80th Percentile
- Funding Above Inflection Points
- Why Budget to the 70th or 80th
- Inflection Points for Lognormal
- Issues with Percentile Funding
- ldquoHere There Be Dragonsrdquo
- Distribution Comparison
- Distribution Comparison Tail Behavior
- Distribution Comparison ndash Tail Behavior (Table)
- Percentile Budgeting is Not Risk Management
- Joint Confidence Level Analysis
- Schedules Impact Cost
- The Relationship of Cost to Schedule
- The Impact of Schedule GrowthCompression on Cost
- Schedule as an Independent Variable
- Schedule is a Dependent Variable
- Incorporation of Schedule in Cost Risk
- Subjective Cost Risk Methods and Overinclusion of Risk Factors
- Fundamental Equation of Cost and Schedule Interrelationships
- How to (Correctly) Incorporate Schedule into Cost Risk Analysis
- Joint Lognormal and Normal Distributions
- Modeling Cost and Schedule as a Bivariate Distribution
- Bivariate Lognormal Distribution for Cost and Schedule
- Bivariate Lognormal for Cost and Schedule
- Correlation Between Cost and Schedule
- Correlation Between Cost and Schedule Caveats
- Conditional Confidence Levels
- Joint Confidence Levels
- Joint Confidence Levels Versus Cost Confidence Levels Only
- Joint Cost and Schedule Confidence
- Cost and Schedule Trade-offs for a Given Level of Joint Confidence
- Policy Considerations for Confidence Levels
- Copulas
- Parametric Joint Cost and Schedule Analysis
- Parametric Joint Cost and Schedule Analysis (2)
- Resource-Loaded Schedules
- Resource-Loaded Schedules Vs Parametrics
- Resource-Loaded Schedules
- Execution ndash Cross Checks
-