Wednesday 25 Nov 2020 | 02:03 | SYDNEY
Wednesday 25 Nov 2020 | 02:03 | SYDNEY

The last word on intelligence agencies


Sam Roggeveen


11 January 2008 13:07

With this rejoinder from Frank Ashe, I'll close down the intelligence discussion for now:

Your position in the last post on intelligence is too pessimistic. Going further into questions of risk management and using the terminology of Tetlock, I'd make the following points:

  • Experts perform no better than dilettantes, but what Tetlock defines as a dilettante is not your typical man in the street. A dilettante is an expert in one area, say the Middle East, being asked for a prediction on the Korean peninsula. The dilettante is highly intelligent and well read, but knows they are not an expert in that area and so tends to make non-extreme predictions with lower degrees of confidence than the expert. The expert will tend to make more extreme predictions and have greater confidence. Extreme events tend not to happen as often as experts predict they will and so the dilettante ends up being as good a predictor.
  • One way of mitigating the effect of extreme experts is to ensure a diversity of opinion enters the analysis and has the possibility of an effect on the outcome. The Roman Catholic Church, for instance, has institutionalised this process with the appointment of a Devil's Advocate in the canonisation process. Heuer's book, in the concluding chapter, looks at ways to improve of the intelligence analysis. One important point is that a number of hypotheses need to be tested — you could interpret a hypothesis as a prediction — and the mechanism for testing is to look for information that refutes an hypothesis.  The best hypothesis may not be the one with the most information in support (which just raises confidence to possibly too high a level) but the one with the least evidence against it. Using your example, I'm sure we could find lots more evidence against a US invasion of Australia than we could, say, for an Indonesian invasion of PNG.
  • Acting on a principle of falsifiability will most probably lead to a number of predictions (hypotheses) with similar degrees of likelihood. By its very nature this outcome will most probably lead to a flexible policy environment. 

So experts do have a valuable place in the intelligence and policy framework, we just need to ensure that the final policy is based on the proper use of their analysis. Many political environments are structured in this way — a set of dilettantes (in Tetlock's sense) are elected to power and their decisions are based on information provided by a wide body of experts. Dangers occur when experts are elected to power or those in power rely on too narrow a field of experts. This brings Karl Popper's The Open Society and Its Enemies to mind — maybe I should go back and reread it.