What do the following high-profile disasters have in common: World War I, Vietnam, the war in Iraq, the collapse of the banking system, and underpreparedness for natural disasters such as Hurricane Katrina?

According to Dominic Johnson at the University of Edinburgh and his pal James Fowler at the University of California, San Diego, the answer is that they have all been blamed on the all-too-human condition of overconfidence.

The puzzle about overconfidence is its ubiquity. Many studies have shown that most people have an exaggerated sense of their own capabilities, an illusion that they have control over uncontrollable events and are invulnerable to risk. Most people, for example, believe they are above-average drivers, a statistical impossibility. We are all overconfident in one way or another.

But how can such a condition have evolved when overconfidence can lead to destruction of communities and catastrophic loss of life?

That’s a mystery that many experimental psychologists have wrestled with, but now Johnson and Fowler say they have the answer. By creating a mathematical model of the way overconfident individuals compete against ordinary individuals, they show that there is a clear advantage in overconfidence.

In fact, if the potential reward is at least twice as great as the cost of competing, then overconfidence is the best strategy. In fact, overconfidence is actually advantageous on average, because it boosts ambition, resolve, morale, and persistence. In other words, overconfidence is the best way to maximize benefits over costs when risks are uncertain.

That’s an interesting insight. Experimental psychologists have long known of the role of overconfidence in conflict situations and yet have been unable to explain its origin.

But it is Johnson and Fowler’s predictions that are most worrying. Their model implies that optimal overconfidence increases with the magnitude of uncertainty. So the greater the risk, the more overconfident individuals should become.

Johnson and Fowler use that finding to predict that overconfidence will be particularly prevalent in domains where the perceived value of a prize sufficiently exceeds the expected costs of competing.

What might these domains be? Johnson and Fowler highlight several, but perhaps the most obvious and potentially dangerous are international relations, where events are complex and distant and involve foreign cultures and languages; new technologies such as the Internet bubble; and the banking industry, where complex financial instruments abound. Any of that sound familiar?

All of this sets the stage for the next question: how best to mitigate the worst side-effects of rampant overconfidence in a society with a dramatically exaggerated sense of its own abilities.

Ref: arxiv.org/abs/0909.4043: The Evolution of Overconfidence