Dunning-Kruger & Planning Fallacy
The Dunning-Kruger effect and the planning fallacy are two biases that distort how engineers assess their own abilities and predict how long things will take. Dunning-Kruger means beginners overestimate their competence and experts underestimate theirs. The planning fallacy means people consistently underestimate how long tasks will take, even when they know from experience that they always underestimate. Both are well-documented, experimentally replicated, and deeply resistant to awareness alone. Knowing about these biases does not make you immune to them.
The Dunning-Kruger Effect
David Dunning and Justin Kruger published their research in 1999. They found that people with low skill in a domain consistently overestimate their performance, while people with high skill consistently underestimate theirs. This is not about intelligence — it is about the relationship between skill and self-assessment.
The Dunning-Kruger curve:
Confidence
^
| * (Peak of Mount Stupid)
| *
| *
| *
| *
| * (Valley of Despair)
| *
| *
| * * * * * (Slope of Enlightenment)
| * (Plateau of Competence)
+--------------------------> Expertise
Beginner: high confidence, low ability
Intermediate: low confidence, growing ability
Expert: calibrated confidence, high ability
The mechanism is simple: to know how bad you are at something, you need the same skills that make you good at it. A junior engineer who does not understand distributed systems cannot accurately assess how hard a distributed systems problem is. They lack the knowledge to know what they do not know.
Engineering Manifestations
The junior engineer:
"I can build that in a week"
What they do not know they do not know:
- Edge cases in authentication
- Database migration complexity
- Error handling in distributed calls
- Backward compatibility requirements
- Testing strategy for concurrent operations
Actual time: 3-4 weeks
The senior engineer:
"This is really hard. I would estimate 3-4 weeks."
What they know they do not know:
- Exact complexity of the third-party integration
- Potential performance issues under load
- Unknown dependencies that will surface
They account for uncertainty. Their estimate is closer
to reality, but they feel less confident about it.
Dunning-Kruger in Technical Discussions
Code review:
Junior reviewer: Approves quickly, finds no issues
(They cannot see problems they do not understand)
Senior reviewer: Takes longer, finds subtle issues
(They recognize patterns from experience)
The junior reviewer's quick approval feels efficient.
The senior reviewer's slow review feels like over-thinking.
But the senior is catching real problems.
Architecture decisions:
Engineer with 1 year of experience:
"Let us use event sourcing, CQRS, and microservices"
(Confident because they read about it, unaware of
operational complexity)
Engineer with 10 years of experience:
"Let us start with a monolith and extract services
when we have clear boundaries"
(Less exciting but aware of the operational cost of
premature distribution)
Technology evaluation:
After reading a tutorial:
"Rust is easy, we can rewrite everything in Rust"
(Dunning-Kruger: the tutorial covered 5% of what
a production Rust codebase requires)
After using Rust for 2 years:
"Rust is powerful but the learning curve is steep.
A rewrite would take 6-12 months with significant
risk during the transition."
The Valley of Despair Is Productive
The intermediate phase — where you realize how much you do not know — feels terrible but is actually the most productive learning period.
Junior phase: "I know how to code!"
Reality: Can write basic CRUD applications
Feeling: Confident
Intermediate phase: "I know nothing"
Reality: Understanding complexity of production systems
Feeling: Overwhelmed, impostor syndrome
This is where real growth happens.
Senior phase: "I know what I know and what I do not know"
Reality: Can estimate complexity, identify risks,
design for failure modes
Feeling: Calibrated confidence
Organizational Dunning-Kruger
Teams and organizations exhibit Dunning-Kruger too.
Young startup:
"We will build our own database, payment system,
and ML pipeline. How hard can it be?"
(Team has never operated these systems at scale)
Mature company:
"We will use managed services for infrastructure
and focus our engineering on our core domain."
(Team has been burned by building undifferentiated
infrastructure and knows the hidden costs)
The startup's confidence is not a strength.
It is Dunning-Kruger at the organizational level.
The Planning Fallacy
The planning fallacy, identified by Kahneman and Tversky in 1979, is the systematic tendency to underestimate the time, cost, and risk of future actions while overestimating their benefits. It persists even when people have direct experience with similar tasks taking longer than expected.
The planning fallacy cycle:
1. Estimate task: "2 weeks"
2. Task actually takes: 5 weeks
3. Post-mortem: "We underestimated. Next time we will
add buffer."
4. Next estimate with buffer: "3 weeks" (added 50%!)
5. Next task actually takes: 4.5 weeks
6. Repeat forever.
The buffer is never enough because the planning
fallacy affects the buffer estimate too.
Why the Planning Fallacy Persists
Reason 1: Best-case thinking
When estimating, engineers imagine the happy path:
- No unexpected bugs
- No dependencies delayed
- No requirements changes
- No context switching
- No sick days or holidays
Each of these is individually unlikely to cause problems.
But the probability that NONE of them cause problems
is very low.
Reason 2: Ignoring historical data
Engineer's track record:
Task 1: Estimated 3 days, took 7 days
Task 2: Estimated 1 week, took 2.5 weeks
Task 3: Estimated 2 weeks, took 4 weeks
Average overrun: 2.2x
Engineer's next estimate: "2 weeks"
(Not "2 weeks * 2.2 = 4.4 weeks")
They know they always underestimate.
They still underestimate.
Reason 3: Decomposition illusion
Breaking a task into subtasks makes each subtask
feel more estimable. But each subtask estimate has
the same bias, and the errors compound.
5 subtasks, each underestimated by 30%:
Total underestimate: not 30% but potentially 50-80%
because delays in subtasks cascade.
Engineering Examples of the Planning Fallacy
"Quick" bug fix:
Estimate: "30 minutes, it is a simple fix"
Reality: 30 minutes to find the bug, 2 hours to
understand the surrounding code, 1 hour to fix without
breaking other things, 1 hour to test, 30 minutes
for code review. Total: 5 hours.
Migration project:
Estimate: "3 months to migrate to the new API"
Reality:
Month 1: Start migration, discover 40% of endpoints
have undocumented behavior
Month 2: Reverse-engineer undocumented behavior,
update migration plan
Month 3: Complete 60% of migration, realize data
format differences require a compatibility layer
Month 4-5: Build compatibility layer, complete migration
Month 6: Fix bugs discovered in production
Actual: 6 months (2x estimate)
Sprint velocity:
Sprint 1: Committed to 30 points, delivered 18
Sprint 2: Committed to 25 points, delivered 16
Sprint 3: Committed to 25 points, delivered 20
Sprint 4: Committed to 28 points
(Why 28? Velocity is clearly 16-20. The planning
fallacy makes you commit to more than you can deliver.)
Where the Extra Time Goes
Hidden time sinks that estimates miss:
Meetings and context switching: 15-20% of the week
Code review (giving and receiving): 10-15%
Debugging unexpected issues: 10-20%
Waiting for CI/CD: 5-10%
Requirements clarification: 5-10%
Environment and tooling issues: 5%
On-call interruptions: 5-10% (if on rotation)
Total overhead: 55-90% of the week
If you estimate "2 weeks of focused coding,"
you actually have 2-5 days of focused coding
in a 2-week period. The estimate should be 4-5 weeks.
Combining the Two Biases
Dunning-Kruger and the planning fallacy compound each other. A junior engineer (Dunning-Kruger) who underestimates task complexity will also underestimate duration (planning fallacy), producing estimates that are wrong on both dimensions.
Junior estimate for building an authentication system:
Dunning-Kruger: "Auth is straightforward" (underestimates complexity)
Planning fallacy: "I can do it in 1 week" (underestimates duration)
Combined: 1 week estimate for a 6-week task
Senior estimate for the same task:
Calibrated complexity: "Auth has many edge cases and security
requirements"
Calibrated duration: "3-5 weeks with testing and security review"
Combined: 3-5 week estimate for a 4-week task (much closer)
Practical Countermeasures
For Dunning-Kruger
1. Seek feedback early and often
Do not build for 2 weeks then show your work.
Show a rough prototype after 2 days.
Feedback calibrates your self-assessment.
2. Study failure modes before building
Before implementing, ask: "What are 5 ways this
could go wrong?" If you cannot think of any,
you are in Dunning-Kruger territory.
3. Ask experienced people to estimate complexity
"How hard do you think this is?" from someone with
10 years of experience is a calibration input.
4. Track your estimates vs. actuals
Over time, the data shows you your bias.
A spreadsheet with (estimated, actual) pairs
is a powerful self-correction tool.
For the Planning Fallacy
1. Reference class forecasting
Do not estimate from the inside view.
Look at how long similar tasks took historically.
Use that as your baseline.
2. Multiply your estimate
If your track record shows 2x overruns, multiply
by 2. If 1.5x, multiply by 1.5.
This feels wrong. Do it anyway.
3. Use the three-point estimate
Optimistic: What if everything goes perfectly?
Most likely: What do you actually expect?
Pessimistic: What if multiple things go wrong?
Weighted average: (O + 4M + P) / 6
4. Buffer, then buffer again
Add 30% buffer for "unknown unknowns."
Then add another 20% for the planning fallacy
in your buffer estimate.
5. Track velocity, not commitments
Use historical velocity (what you actually delivered)
not planned velocity (what you hoped to deliver).
The Estimate-Actual Spreadsheet
Keep a running log:
Task | Estimated | Actual | Ratio
--------------|-----------|--------|------
Auth system | 5 days | 12 days| 2.4x
API refactor | 3 days | 5 days | 1.7x
DB migration | 10 days | 18 days| 1.8x
Bug fix batch | 2 days | 4 days | 2.0x
New endpoint | 1 day | 1.5 day| 1.5x
--------------|-----------|--------|------
Average ratio: 1.9x
Your personal correction factor: 1.9x
Next estimate: multiply raw estimate by 1.9
This is the most effective countermeasure for the planning fallacy. Hard data about your own bias is difficult to ignore.
Common Pitfalls
- Believing awareness prevents the bias: Knowing about Dunning-Kruger does not prevent it. Knowing about the planning fallacy does not fix your estimates. You need structural countermeasures: data tracking, peer review, reference class forecasting.
- Over-correcting for Dunning-Kruger: Not all confidence is Dunning-Kruger. Sometimes a junior engineer is right that something is simple. The bias is a tendency, not a law. Use data, not labels.
- Padding estimates instead of improving them: Adding a generic "50% buffer" is not the same as understanding why your estimates are wrong and fixing the root cause (missing hidden work, ignoring overhead, best-case thinking).
- Dismissing junior estimates entirely: Juniors sometimes have creative solutions that seniors overlook. The correction for Dunning-Kruger is calibration, not dismissal. Ask juniors to explain their reasoning, then calibrate.
- Not tracking historical data: Without data on your actual vs. estimated performance, you cannot calibrate. The estimate-actual spreadsheet is simple and transformative.
- Confusing impostor syndrome with calibrated humility: The intermediate engineer who says "I do not know enough" might be experiencing Dunning-Kruger's valley of despair (productive learning) or genuine impostor syndrome (harmful self-doubt). Distinguish between healthy humility and unhealthy self-undermining.
Key Takeaways
- Dunning-Kruger: beginners overestimate, experts underestimate. The junior who confidently says "1 week" and the senior who says "this is hard" — the senior is usually closer to reality.
- Planning fallacy: everything takes longer than expected, even when you account for things taking longer. This bias persists across decades of experience.
- Track your estimates vs. actuals. Calculate your personal correction factor and apply it to every estimate.
- Use reference class forecasting: "How long did similar things take historically?" beats "How long will this specific thing take?"
- Seek calibration from more experienced engineers for complexity assessment and from your own historical data for duration estimation.
- Structural countermeasures (tracking data, peer estimation, three-point estimates) work better than awareness alone.