P95 Probability or Implausibility

For the past 18 months or so I have heard a lot about the P95.

What is a P95?
Good question!

I was told that it was a 95% likelihood that the Waimea dam would be built on or under budget.

Given the current state of press releases coming out of the Tasman Council, I thought it would be timely to delve a little deeper into this magical assurance while we await the next announcement. Incidentally, press releases are the only way anyone gets any information regarding the dam (not as big an exaggeration as you might imagine, councillors are just privileged to receive their copy 15 minutes ahead of the general ratepayer).

Back in January 2016, we read in the Nelson Mail the following:

The dam’s estimated cost rose to $83m last year after costs were revised by the council’s consulting engineers…

The council’s $83m figure is based on a risk factor which assumes there is a 95 per cent probability the dam can be built for that amount of money or less. It’s known as P95.

WCD project manager Nick Patterson said the dam can be built for $73m, including the $6.5m already spent. This was using a P50 risk factor that assumed the dam could be built for that amount of money or less.

There was a lot of confidence around what price the dam could be built for and the level of certainty once we added a few extra million and called it a P95. These confident assertions have continued right up until the press release on July 5 2018.

The Mayor was an early adopter of this confident boast as is recorded in articles such as this Stuff article on September 2017:

Richard Kempthorne said the dam costing was based on a “P95” confidence level, which meant the council could have 95 per cent confidence the dam would be constructed at or below the estimated cost.

“At or below the estimated cost” was the catchphrase that echoed the Council chambers every time anyone questioned the possibility of cost overruns. We have used a P95 to arrive at the budgeted figure.

On the 10th of August 2017 Natasha Berkett is quoted in the Farmers Weekly as saying:

However, there was also a $13 million contingency allowance in the $80m project for further increases.

“We are working to a P95 standard that is where we are 95% certain about the cost of the project.”

Waimea Irrigators Ltd advocates John Palmer and Murray King scoffed at any hint that there would be any budget blowout. The Waimea Water site boasts the following:

Are the cost estimates for the Dam accurate?

A great deal of work has gone into the current cost estimates for the dam construction and project leads are currently undertaking another robust analysis.

The base cost estimate for the Dam construction is $50m. On top of that there is $13.5m included in the project estimate for changes in scope and unexpected cost increases. This raises the level of confidence in the estimate to a very high level – there is only a 5% probability that the costs will exceed the $63.6m budgeted and a 95% probability the project will come in on or below budget. That level of confidence is far higher than the typical estimates you might see in other construction projects (emphasis added)

I hope you are feeling convinced in the level of confidence that we have that the dam will come in on or under budget. That is the completed dam, including construction cost blow-outs. We are 95% certain in our facts.

Just in case you still had a few doubts we put out a Dam Straight fact campaign at the ratepayer’s expense of almost fourteen and half thousand dollars (which I realise is chicken feed compared the ten million, in round figures, lost on the project to date).

To prevent large cost overruns, Council is doing all it can to keep risks to a minimum including:

• using a p95 level for construction costs, meaning that there is a 95% probability that the project will come in at or under budget;
• using modern engineering and risk management approaches and external professional advice; and
• $13.5 million contingency for changes in scope and other unforeseen eventualities.

Can there be any doubt as to the robustness of the budget after we used a P95 level, modern engineering and risk management approaches, external professional advice, and even a whopping $13.5m contingency figure for good measure?

Well, some people thought so. One such person is Paul MacLennan who indicated:

TDC’s own recent admission they have never priced the civil cost of the 87 Hectare Reservoir and sealing of the lake bed to hold the 13.4 million cubic meters of water is appalling and makes the TDC claim $76.9 million at P95 a gross misrepresentation.(emphasis added)

A rather strong assertion. He claimed that “clearing 87 Hectares of steep, sloping, and fractured terrain will cost $30 million plus.”  I tried to follow up on his concerns with staff only to receive repeated replies attesting to the robustness of our P95. A P95 that it seems, from reading between press release lines, was so grossly inadequate that it didn’t even cover the quote, never-mind the inevitable construction overruns.

In his submission, during the Waimea Dam consultation process, Dr. Roland Toder questioned the P95 claim.

A so-called ‘P95’, as mentioned by TDC is misleading. In project management, percentiles are expressed by P followed by the percentile. A P95 estimate of  $82.5million means that the project has a 95% chance of being completed with this or less amount of money. Normally one uses data one has from similar activities (like executed several dams of similar size before). Ideally, if you have a number of similar activities (many dams of similar size) that would be best, because from this data, you need to find the mean (expected value) and the standard deviation (to determine the range of error) for each activity. THEN you can calculate cost and timing estimates for the entire project.(emphasis added)

People from the Water Information Network Inc also protested the validity of the P95 figure sold by WIL and Council staff.

• Tasman District Council (TDC) ad­mit they have never priced the clearing cost of the 87 hectare reser­voir.
• With steep, slop­ing and frac­tured ter­rain clearance of the area is likely to cost over $30 mil­lion.
• Similarly, no costing has been made for the seal­ing of the lake bed to hold 13.4 mil­lion cu­bic me­tres of wa­ter.
• Removed trees, bushes, shrubs, de­tri­tus, top­soil, and stumps will add up to a me­tre-high pile.
• This will amount to around one mil­lion cubic metres to shift from the Dam’s foot­print area to a stock­pile area where run-off can be man­aged un­til fi­nal dis­posal.
• Silting up risks and decommissioning costs have not been taken into account.
• Suitability of rock fill ‘borrowed’ for the Dam wall has not been accurately assessed. So it may not be suitable.
• All of this indicates that TDC’s claim of $76.9 mil­lion at p95 (95% ‘accurately’ costed) is a gross mis­rep­re­sen­ta­tion. (emphasis added)

I repeat that when I questioned the validity of these claims I was told that we had sufficiently budgeted all aspects of the dam, and early indications on pricing confirmed the validity of the budget.

When I questioned the age of the original estimate given that all our current contracts are experiencing a premium in a high demand marketplace I was again assured that the estimated budget for the Dam was adequate.

At the councillor 15-minute-advance-notice-of-the-press-release meeting, I asked TDC engineering department head Richard Kirby which aspects of the dam quote were currently running over budget he said he didn’t know. When asked what was the range of budget blowout on the figures that had come in to-date he responded that he didn’t know. When I questioned that they had called an extraordinary briefing and are about to send out a press release announcing that there are significant challenges with the contract price meeting budget and yet you are telling me that you have no idea what the overrun gap is? He confirmed that was correct.

The head of Engineering doesn’t know the figures.

The Mayor doesn’t want to know the figures.

I want to know whose head is going to roll for misleading the Council over the 95% certainty of the price for building the Waimea Community Dam. Information that resulted in the Council spending the best part of ten million dollars of ratepayer’s money progressing the project to this point. Whether deliberately misleading councillors or doing so through incompetence someone has to be held accountable.