Are My Project Forecasts Accurate?

Mean Absolute Percentage Error / Mean Percentage Error assess EVM time and cost forecast method accuracy. 3 min read.

EVM Forecast Accuracy Methods

To evaluate time and cost forecast accuracy, the error of the forecast along all review periods needs to be measured. For each review period, the forecasting error is the deviation between the current duration and cost forecast value versus the final duration and cost outcome.

Mean Absolute Percentage Error (MAPE)

The Mean Absolute Percentage Error (MAPE) is the average of the absolute percentage errors overall review periods r = 1, …, R. Obviously, the lower the MAPE value, the more accurate the prediction.

Mean Percentage Error (MPE)

The Mean Percentage Error (MPE) is the average of percentage errors based on total errors regardless of sign. MPE values lower than zero denote an underestimation while MPE values above zero refer to overestimations. However, since the average process sums up positive and negative percentage errors, they tend to cancel each other out, resulting in unrealistically low MPE values.

EVM Forecast Accuracy

The time / cost forecasting accuracy calculations are given using the following,

EVM Forecast Method Drawback

A significant drawback of the accuracy metrics is that their value is only known when the project finishes, and as such, this knowledge is of little value except for similar future projects. However, the MAPE and MPE metrics can be calculated ex-ante, before the start of the project, using predefined activity duration and cost distributions and Monte Carlo simulations.

These simulations imitate the fictitious progress of the project by generating random variations in both activity durations and costs under a designed experiment. These fictitious project executions result in time-based values for the actual cost (AC) and earned value (EV), which are translated into EVM performance and forecasting metrics.

After each simulation run, the final project duration and cost outcomes are known, and the resulting EAC(t) and EAC($) metrics are then evaluated using the MAPE and MPE metrics. This knowledge is then available before the project starts and is used as a selection tool for the most accurate forecasting method.

Microsoft Project Demonstration Project

Shown below is a Microsoft Project demonstration project, which is adequately resourced and costed. This project has a schedule duration of 9-weeks and a cost budget of $150. The delivered project and the resulting tracking information, show a final duration of 11-weeks and an actual cost of $210.

This slideshow requires JavaScript.

EVM Forecast Accuracy Results

Shown below are the time and cost MAPE / MPE results,

This slideshow requires JavaScript.

EVM Forecast Accuracy Conclusions

Comparing each of the results, it is clear that all methods over the life of the project are highly accurate. However, closer examination shows highly inaccurate results during the first third of project progress, which only begins to stabilise when the project is half-way delivered. Therefore, to build project team, management, and customer confidence, it is better to select the most accurate upper bound (overestimation) and lower bound (underestimation) method, which will provide actionable data-insights.

Unfortunately, conclusions drawn for this forecast accuracy study can’t be generalised because each project is unique, and characteristics from one project to another differ significantly.

If you found this article informative, then please show your appreciation by liking this post.

Learn More

If you would like to know more about leveraging data-driven actionable insights for your project schedule, then feel free to contact me on itierney@pminsight.com.au.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.