As a software developer (Architect?) I find myself in a constant battle with clients and project managers about changing estimates and their inaccuracy. I've lost count the number number of times I've given an estimate and then had to revise it to the dismay of folks who assumed seemingly subtle changes would move the estimate down or allow it to remain the same, only to see it creep up. There are a number of variables that increase the estimation risk and I'll briefly touch on the major factors.
Major factors are:
- Changing requirements/assumptions - a fundamental truth is that any change incurs overhead. The act of changing the design to remove a requirement is is work too...remember, even simplifying the design is work. Removing a requirement mandates revalidating a design against (even if they're simplified) the new requirements.
- Changing the team structure - an experienced dev is much more effective than a newbie. Moreover a person well versed in a particular solution is often more effective than a more experienced resource who is unfamiliar with the existing code. Creating estimates for an unknown team is tremendously difficult and often leads to large inflations to account for the risk of getting an unvetted resource.
- Work fills to accomodate available time - if you give a developer 40 hours to complete a task, they will almost always take AT LEAST that amount of time. Even if it seems to be a simple task, they will spend extra time to analyze options, test, and otherwise use available time even if they COULD have potentially "just done the work" in four hours.
- Estimates are just starting points - the harsh reality is that estimates for non-trivial software are starting points and evolve as more information becomes available. The more analysis you do without obtaining more information, the higher the multiplier is that the estimate is based on faulty information (especially when it involves the factors mentioned above
The short version is that "all software development is design". Any change anywhere changes that design and thus creates more work. Agile proponents realize this (maybe implicitly) and combat this problem by locking down design for periods of time to help move things forward (with real deliverables). Long drawn out design cycles cause extra work that too often is underwater.