Bart Blog

Why tech projects almost always take longer than anyone expects – and what that means for our digital future

It’s a scene that plays out in planning meetings across the tech industry with depressing regularity: A project timeline appears on screen, filled with carefully plotted milestones, as someone confidently declares that the new software project will launch in six months. Fast-forward a year later, and that same project is still in development, the budget has doubled, and half the original team has moved on to other companies.

I’ve witnessed this pattern countless times throughout my career, and it prompted me to dig deeper: Are these just unfortunate coincidences, or is there something more systematic at play? The data confirmed my suspicions- and then some.

This isn’t just anecdotal evidence of poor planning – it’s the predictable outcome of one of the tech industry’s most persistent and costly problems. When it comes to estimating how long software projects will take, we’re not just bad at it. We’re catastrophically bad at it. As physicist Niels Bohr once observed, “Prediction is very difficult, especially about the future”-and nowhere is this more apparent than in software development.

The Numbers Don’t Lie

The data is staggering. According to the Standish Group’s landmark research, only 16% of software engineering projects are completed on time and within budget. Let that sink in: In an industry that prides itself on precision and efficiency, more than eight out of ten projects fail to meet their basic deadlines.

But it gets worse. The same research found that the average project schedule overrun is 120% meaning the typical software project takes more than twice as long as originally estimated. This isn’t a matter of being off by a few weeks; it’s systematic, chronic underestimation on a massive scale.

Harvard Business Review’s analysis of IT projects paints an equally grim picture. Their research revealed that one in six projects are late by almost 70%, while some individual studies have documented cases where projects took 700% longer than estimated, nearly eight times the original timeline.

Why We Keep Getting It Wrong

The problem isn’t that software developers are uniquely bad at estimation. Rather, it’s that software development is uniquely resistant to accurate prediction. Unlike building a bridge or manufacturing a car – where processes are well-understood and variables can be controlled – software projects involve solving problems that have never been solved before, using technologies that are constantly evolving, for requirements that shift as stakeholders better understand what they actually need.

The research identifies several key factors that consistently derail project timelines:

Incomplete requirements: Nearly half of all project failures stem from unclear or changing specifications. Projects begin before anyone fully understands what needs to be built.

Unrealistic time frames: The pressure to deliver quickly leads to estimates based more on wishful thinking than engineering reality.

Technology challenges: New tools and platforms promise efficiency gains but often introduce unexpected complexity.

Team dynamics: Communication breakdowns and coordination problems multiply as team sizes grow.

The Compound Effect

Perhaps most troubling is the finding that every additional year in a project’s planned duration increases the likelihood of overruns by 15%. This suggests that our estimation problems aren’t just about individual tasks-they compound over time, making longer projects exponentially more likely to fail.

This helps explain why large-scale software transformations, the kind that promise to revolutionize entire industries, so often become cautionary tales. The more ambitious the vision, the more likely it is to collapse under the weight of unrealistic expectations.

The Uncertainty We Can’t Escape

The implications extend far beyond the tech industry. As software becomes embedded in everything from healthcare to transportation to finance, these estimation failures don’t just delay product launches, they can affect critical infrastructure and public services. And as our digital ambitions grow ever grander, from autonomous vehicles to artificial intelligence to smart cities, this problem is only becoming more consequential.

What Actually Works

While the statistics paint a grim picture, research has identified specific interventions that can dramatically improve project outcomes. The most effective approaches don’t just reduce delays – they transform how software gets built.

Keep Teams Small: Studies analyzing nearly 2,000 IT projects found that teams of 3-7 people achieve the highest productivity. Beyond 9 members, productivity measurably declines. Small teams can save up to 25% in costs while delivering faster results.

Break Projects Into Short Cycles: Organizations that divide large projects into shorter delivery cycles see substantial improvements. One approach requires breaking all activities into chunks of fewer than 20 person-days that take no longer than four weeks to complete.

Fix Requirements Early: Since incomplete and changing requirements cause the majority of overruns, successful organizations spend significant effort in a focused requirements phase, often six months, with all stakeholders signing off before development begins. After that, changes require board-level approval.

Use Data-Driven Estimation: Teams that track their historical performance and use machine learning-enhanced estimation techniques achieve 30-40% better accuracy than those relying on gut instinct. The key is building institutional memory of how long similar work actually takes.

Deploy Continuously: Elite-performing organizations deploy code multiple times per day compared to once per week for average teams. This approach, supported by automated testing and deployment pipelines, reduces risk and provides faster feedback on progress.

Perhaps the issue isn’t that we’re bad at estimation, but that we’re trying to impose false precision on an inherently uncertain process. As economist John Kenneth Galbraith noted, “The only function of economic forecasting is to make astrology look respectable.” The same might be said of software project estimation.

But unlike economic forecasting, we have concrete evidence of what works. The organizations that consistently deliver software on time don’t just hope for better estimates, they redesign their entire approach to development. In the world of software development, uncertainty isn’t a bug, it’s a feature we’ve need to fully accept.

---

Sources: Standish Group CHAOS Reports, Harvard Business Review IT Project Studies, McKinsey Research, Quantitative Software Management Team Size Studies, “Empirical findings on team size and productivity in software development”, “Software Effort Estimation Accuracy Prediction of Machine Learning Techniques”, DORA Metrics Research, “Why is Software Late? An Empirical Study of Reasons for Delay in Software Development”