Tuesday, December 05, 2006

Part II: A Failure of Intelligence

Prominent physicist Freeman Dyson recalls the time he spent developing analytical methods to help the British Royal Air Force bomb German targets during World War II.

By Freeman Dyson

Another job that Smeed gave me was to invent ways to estimate the effectiveness of various counter­measures, using all the evidence from a heterogeneous collection of operations. The first countermeasure that I worked on was MONICA. MONICA was a tail-mounted warning radar that emitted a high-pitched squeal over the intercom when a bomber had another aircraft close behind it. The squeals came more rapidly as the distance measured by the radar became shorter. The crews disliked MONICA because it was too sensitive and raised many false alarms. They usually switched it off so that they could talk to each other without interruption. My job was to see from the results of many operations whether MONICA actually saved lives. I had to compare the loss rates of bombers with and without MONICA. This was difficult because MONICA was distributed unevenly among the squadrons. It was given preferentially to Halifaxes (one of the two main types of British heavy bomber), which usually had higher loss rates, and less often to Lancaster bombers, which usually had lower loss rates. In addition, Halifaxes were sent preferentially on less dangerous operations and Lancasters on more dangerous operations. To use all the evidence from Halifax and Lancaster losses on a variety of operations, I invented a method that was later reinvented by epidemiologists and given the name "meta-­analysis." Assembling the evidence from many operations to judge the effectiveness of MONICA was just like assembling the evidence from many clinical trials to judge the effectiveness of a drug.

My method of meta-analysis was the following: First, I subdivided the data by operation and by type of aircraft. For example, one subdivision would be Halifaxes on Bremen on March 5; another would be Lancasters on Berlin on December 2. In each sub­division I tabulated the number of aircraft with and without MONICA and the number lost with and without MONICA. I also tabulated the number of MONICA aircraft expected to be lost if the warning system had no effect, and the statistical variance of that number. So I had two quantities for each subdivision: observed-minus-expected losses of MONICA aircraft, and the variance of this difference. I assumed that the distributions of losses in the various subdivisions were uncorrelated. Thus, I could simply add up the two quantities, observed-minus-expected losses and variance, over all the subdivisions. The result was a total observed-minus-expected losses and variance for all the MONICA aircraft, unbiased by the different fractions of MONICA aircraft in the various subdivisions. This was a sensitive test of effectiveness, making use of all the available information. If the total of observed-minus-expected losses was significantly negative, it meant that MONICA was effective. But instead, the total was slightly positive and less than the square root of the total variance. ­MONICA was statistically worthless. The crews had been right when they decided to switch it off.

I later applied the same method of analysis to the question of whether experience helped crews to survive. Bomber Command told the crews that their chances of survival would increase with experience, and the crews believed it. They were told, After you have got through the first few operations, things will get better. This idea was important for morale at a time when the fraction of crews surviving to the end of a 30-­operation tour was only about 25 percent. I subdivided the experienced and inexperienced crews on each operation and did the analysis, and again, the result was clear. Experience did not reduce loss rates. The cause of losses, whatever it was, killed novice and expert crews impartially. This result contradicted the official dogma, and the Command never accepted it. I blame the ORS, and I blame myself in particular, for not taking this result seriously enough. The evidence showed that the main cause of losses was an attack that gave experienced crews no chance either to escape or to defend themselves. If we had taken the evidence more seriously, we might have discovered Schräge Musik in time to respond with effective countermeasures.