What is estimation error

The Errors of Estimation

What we often hear from estimators is that estimates are supposed to be “intervals”, not single-point values.

This blog post is now a podcast also, narrated by Billy Elchin

As usual, they don’t define what “interval” should be. They just say that you have to be “accurate”, but not so “precise”.

Ok, let’s explore what that might mean.

Let’s say you get a request to estimate a project. You do your homework, and you come up with a number. Say: 20 “man-days”. What does that mean? Does it mean that you can deliver it in 1 day if you have 20 people working on it?

Surely that’s not the case, or even expected. Just think of testing, UX work, development, etc.

So, the estimate will be more something like:

• 3 man-days UX design
• 5 man-days of PM work
• 7 man-days of Dev work
• 4 man-days of testing
• 1 man-day of release and shipping related tasks

For a total of, you guessed it, 20 man-days.

Interval magic

OK. Now, let’s go back to that interval we were talking about. Let’s say that your margin (estimators also call it “contingency”) is 20%. In that case the estimate could be something like:

• 3 to 3.6 man-days UX design
• 5 to 6 man-days of PM work
• 7 to 8.4 man-days of Dev work
• 4 to 4.8 man-days of testing
• 1 to 1.2 man-day of release and shipping related tasks

Now we have the interval we were talking about above, and it comes out at: [20 to 24] man-days.

Now, some people might take issue with that, and say that the “margin” is in both directions, not just on the “plus” or right-side of the original estimate.

This is where things get complicated, and people will talk about optimistic, pessimistic and “most-likely” estimates (3 point estimation). Without making this too much of an estimate lesson, what that would mean in practice is that everyone in the team must give 3 estimates for every task, and from that calculate, mathematically, what the estimate should be.

The incoherence in estimation practices

However, there’s a problem here.

We started this post by talking about the idea that estimates should be intervals. What happens is that the 3-point estimates drive teams to present a “single-point” estimate for each task that is based on those 3 estimates. I’d say that, logically, asking for a 3 point estimate and then calculating a single point defeats the whole point of having a 3 point estimate!

That’s what we would call an internal incoherence (there’s plenty of those in estimation).

For now, let’s get back to the interval to incorporate the idea that estimates could be wrong in both directions (early and late), but most likely to be late (this we can easily observe in the wild). So, our intervals might be something like:

• 2.8 to 3.36 man-days UX design
• 4.7 to 5.64 man-days of PM work
• 6.5 to 7.8 man-days of Dev work
• 4.7 to 5.64 man-days of testing
• 0.9 to 1.08 man-day of release and shipping related tasks

And now our interval is: [19.6 to 23.52] man-days.

So, now we have 3 possible estimates to give out:

• a) 20 man-days single point estimates.
• b) [20 to 24] man-days.
• c) [19.6 to 23.52] man-days.
Читайте также:  Steam error please launch from steam client

Which one should we use? Wait! We’re not done yet!

Not so fast, what about the fat tails?

There’s another problem in the use of intervals in estimation.

Although the estimators argue that 20% margins are “normal” and “acceptable” (I take issue with that, but that’s another post), the fact is that estimates are more often wrong on the right side (late), than on the left side (early).

So, we should factor that in. But a question arises: how much can an estimate be wrong on the right side (late)?

There’s quite a lot of data on this, but I’ll use a conservative estimate. Let’s say that the estimates can be 250% (2.5 times) wrong on the right side (late).

Is this conservative? Yes. Let’s look at evidence A, from Steve McConnel’s (an estimator and author on how to improve estimates) book “Demystifying the black art of estimation”.

In that book, McConnel shares a graph of the “on-time” record for one of the companies he consulted (adapted here from the NoEstimates book):

Figure 1 – Graph adapted from Steve McConnell’s book: Software Estimation, Demystifying the Black Art.

In Fig. 1, we see those two projects on the top left, which should have lasted around 2 to 7 days, but lasted 200+ days. That’s an error of 2800+%.

In my model, based on data I collected from projects where I worked, the largest error I saw was 250%, so I’ll use that *conservative* margin and apply it to our interval, which gets us to:

• 2.8 to 7 man-days UX design
• 4.7 to 11.75 man-days of PM work
• 6.5 to 16.5 man-days of Dev work
• 4.7 to 11.75 man-days of testing
• 0.9 to 2.25 man-day of release and shipping related tasks

And now our interval is now: [19.6 to 49.25] man-days.

The takeaway

In this example, we’ve established:

a) Intervals are hard to calculate (we’ve explored 3 different ways, that deliver 3 different values), and leave a lot to be interpreted, which defeats the purpose of estimation since every person can choose (and be in line with literature!) whatever method they want.

b) The traditional “margin” of 20% is not applicable in real-life cases, we investigated how even estimation proponents often show data where projects are orders of magnitude late compared to their original estimate (and use single-point to make their point, an internal incoherence).

c) Estimating according to “best practice”, is often a time consuming, and still error-prone practice (e.g. three-point estimate).

d) When we incorporate errors seen in real life, we are often talking about margins that are 10x larger (250%) than what estimators say is acceptable “margin” (usually around 20%)

Estimation is an internally incoherent practice, that often yields information that is inadequate for decision making (would you book 19.6 man-days or 49.25 man-days for this project?).

And on top of that, it increases the effort the teams need to spend in order to deliver software (estimation effort has a cost that – depending on your techniques – might not be irrelevant).

Furthermore, estimation requires us to suspend our belief that Agile approaches are better for software development. For example, you need all the requirements up-front to even do the kind of simple estimation we used in this example.

In short, estimation is a failure-mode in SW development.

Use #NoEstimates instead, and learn what Carmen did to save her project by using #NoEstimates approaches in a project that seemed doomed: The NoEstimates Book.

Источник

How to find standard error of estimate?

1. Calculate the error of each predicted value. In the fourth column of your data table, you will calculate and record the error of each predicted value.
2. Calculate the squares of the errors. Take each value in the fourth column and square it by multiplying it by itself.
3. Find the sum of the squared errors (SSE). .

How to calculate maximum error?

You can use the calculator to compute the MOE in four simple steps: Use the drop-down menu to select the confidence level Input the sample size and then the proportion percentage Click on the «Calculate» button to generate the outputs.

What is the value of the standard error of estimate?

where:

• y: The observed value
• ŷ: The predicted value
• n: The total number of observations

What is the standard error of measurement?

standard error of measurement (SEM), the standard deviation of error of measurement in a test or experiment. It is closely associated with the error variance, which indicates the amount of variability in a test administered to a group that is caused by measurement error.

Читайте также:  Kms server service error

What is meant by estimation error?

The difference between an estimated value and the true value of a parameter or, sometimes, of a value to be predicted.

What type of error is estimation?

Overview. Conceptually, there are three major types of estimating error. These include quantity errors, rate errors, and errors of omission. Most companies underestimate how much these errors are costing them.

What is estimation error in deep learning?

Or in other words, estimation error estimates how good is the algorithm that chooses f from F given training dataset, approximation error estimates how good the function family is. Cite. Follow this answer to receive notifications.

What are the two types of estimation error?

The total error of the survey estimate results from the two types of error: sampling error, which arises when only a part of the population is used to represent the whole population; and. non-sampling error which can occur at any stage of a sample survey and can also occur with censuses.

What is the source of error in estimating?

Both kinds of error may enter into the selection of the observations and into the values (eg integrator readings) of the observations. Systematic errors are those that cause a bias. Acoustic measurements are often affected by a number of biases that can be either additive, or may combine so as to cancel one another.

What are 5 types of errors?

What are the different types of errors in measurement?Constant error. Constant errors are those which affect the result by the same amount. . Systematic error. . Random error. . Absolute error. . Relative error. . Percentage error.

What is approximation error and estimation error?

If we let the function class be large enough to contain the optimal model, then the approximation error can be zero, but on the other hand, the estimation error increases as the larger the function class is, the less likely the algorithm is to find the best model in the class.

What is prediction error in statistics?

In statistics, prediction error refers to the difference between the predicted values made by some model and the actual values. Prediction error is often used in two settings: 1. Linear regression: Used to predict the value of some continuous response variable.

What is training error in machine learning?

Training error is the prediction error we get applying the model to the same data from which we trained. Training error is much easier to compute than test error. Train error is often lower than test error as the model has already seen the training set.

What are the types of errors in statistics?

Two potential types of statistical error are Type I error (α, or level of significance), when one falsely rejects a null hypothesis that is true, and Type II error (β), when one fails to reject a null hypothesis that is false.

Why is estimation important in statistics?

In statistics, estimation allows us to understand the behavior of a population with the help of a small sample.

What is a systematic error?

Systematic error means that your measurements of the same thing will vary in predictable ways: every measurement will differ from the true measurement in the same direction, and even by the same amount in some cases.

Why are actual costs higher than estimated costs?

Other reasons that actual costs may exceed estimated costs include: Unclear or incomplete plans and specifications: The absence of clear plans leaves much room for disagreement about what, exactly, was bid on. This can lead to change orders and extra costs for extra work.

What are the most common causes of disputes in building and remodeling projects?

Cost overruns are probably the most common cause of disputes in building and remodeling projects. The main types of estimating errors are: Omissions: These are items accidentally left out of the estimate – either soft costs (permits, fees, etc.) or hard construction costs. Omissions may be due to items missing from the plans and specs that were, .

What is cost overrun in construction?

Cost overruns from estimating errors are an unfortunate fact of life in construction. There are many ways for jobs to go over budget – that is, cost you more than you or your contractor (or sub) estimated. While some of items discussed below are not typically considered “estimating errors,” all can drive up costs beyond the estimate or bid price. That means someone needs to come up with additional money at the end of the job. Either the contractor makes less profit or you have to come up with extra cash. Who pays for cost overruns is largely determined by your contract.

Читайте также:  Curl error ssl23 get server hello unknown protocol

Can you forget construction costs?

As for soft costs, don’t forget all your permits and fees, which can add up to many thousands of dollars. Then there are the not-so-obvious costs like temporary power, dumpsters, and site prep.

What are the three types of estimating errors?

Conceptually, there are three major types of estimating error. These include quantity errors, rate errors, and errors of omission. Most companies underestimate how much these errors are costing them. Those that ignore best practices for estimating typically fall victim to the winner’s curse – repeatedly “winning” money-losing jobs.

Why is estimating important?

A good estimating system closes the loop. Accuracy improves over time if it includes feedback from purchasing, production, and/or the field to improve it. Creating a feedback loop also helps prevent new staff that is training on the new system from replicating the same error repeatedly while no one is watching.

Why is it important to reduce all three types of errors at the same time?

Reduce all three kinds of error at the same time. Most important is that the effort should be applied to reduce all three errors together rather than trying to improve only one category of error at a time. The reason for this is that changing one area affects the others.

How to prevent errors of omission?

Errors of omission are prevented by different methods in different contexts. Consider creating standardized forms and checklists for various types of jobs or components , and have them include all of the likely components, so that they cannot be forgotten. Also, maintain a database of standard estimates that have been thoroughly reviewed for accuracy to use as templates in creating new estimates. In this fashion, only the modifications can include item errors. Using details from past jobs is also a helpful tool in ensuring that there are no omission errors.

What is the difference between accurate bidder and inaccurate bidder?

The accurate bidder only has bids that fall in the dark blue range, while the inaccurate bidder falls into both the dark and light blue ranges. Scenario 1: The inaccurate bidder has bid way too low and lost money. Scenarios 2 & 3: Both bidders fall in the range of the accurate bidder.

What are the two types of quantity errors?

There are two key types of quantity error: errors on labor and errors on materials , with labor hours typically being the most difficult to estimate. This can be controlled by creating systems for estimators to use in working up their hours.

Is it important to be an estimator?

As everyone knows, sometimes you are high and you win anyway, and sometimes you are just a little low, but it is just enough to win what turns out to be a profitable job.

THE SKILL SET MATCH-UP

To create an initial estimate that’s on-target, project managers must match their team members’ skills to appropriate tasks. This is a particular area of concern, Mr. O‘Brien notes, because many organizations are now operating with fewer staff members.

LIFE IN THE FAST LANE

When estimates are concerned, expect the unexpected, says André Choma, PMP, master engineer at Vale, a mining company based in Belo Horizonte, Brazil. Prior to joining the organization, he worked as a consultant on several projects with aggressive cost and schedule estimates.

WHEN EXECUTIVES DECREE

No matter what rigorous methods and policies an organization applies, it’s impossible to remove the human element from the estimation process, notes Brice Lucas, COO and senior account director at Blue Fountain Media, a website development and online marketing company in New York, New York, USA.

THE DOWNSIDE OF OPTIMISM?

To avoid the project manager blame game, make other departments share accountability for establishing reasonable estimates.

Interval magic

OK. Now, let’s go back to that interval we were talking about. Let’s say that your margin (estimators also call it “contingency”) is 20%. In that case the estimate could be something like:

The takeaway

a) Intervals are hard to calculate (we’ve explored 3 different ways, that deliver 3 different values), and leave a lot to be interpreted, which defeats the purpose of estimation since every person can choose (and be in line with literature!) whatever method they want.

Источник