Wednesday 25 Nov 2020 | 02:40 | SYDNEY
Wednesday 25 Nov 2020 | 02:40 | SYDNEY

Rejoining the intelligence debate


Sam Roggeveen


9 January 2008 13:56

Reader Frank Ashe comments on a debate I started in December (with a response by Paul Monk here) about whether it is really the job of intelligence agencies to make predictions:

Your reference to Tetlock is much needed for a think-tank, any member of which would be designated as one of Tetlock's 'experts' who perform no better than Tetlock's 'dilettantes' in making predictions…one of the most interesting points from Tetlock, and one noticed by many other studies in the psychology of belief, is that the experts have an overestimation of their own predictive ability. As more and more information is available and studied there is a rapid levelling out of predictive ability but no levelling out in the rise in confidence of the accuracy of the prediction. One other attribute of prediction that does not level out - indeed, it can be used to differentiate between expert's predictions and dilettante's predictions - is the rapidity with which defences are wheeled out when a prediction is proven wrong. Experts are much better than dilettantes at coming up with spurious reasons as to why they were actually right!

I’ve done some more thinking about my own position on the role of intelligence agencies, and I’m wondering if it is really tenable.

The thrust of my argument in December was that it was not really the job of intelligence agencies to make predictions because the international political environment is inherently unpredictable, and the best policy-makers can do is design flexible policy to cope with the unknowable, rather than make decisions based on highly unreliable predictions.

But is it even possible to avoid basing policy decisions on predictions? Take defence as an example. It is not enough to just plan for the worst, because we’d then have to build defences that could stop the US from invading, which is plainly unaffordable. Instead, we assume the Americans will stay on our side.

It’s all very well to say, as I did, that policy ought to be flexible in order to cope with unpredictable events, but it can’t be infinitely flexible. Resources are finite, so some unlikely contingencies – like an American invasion of Australia – have to be ruled out. Admittedly, that’s a pretty safe bet, but once you allow one prediction, you have to concede the larger point: it is in fact impossible to make policy without first making predictions. Trouble is, as Tetlock pointed out, experts are not very good at making predictions.

I’m not sure where that leaves me. More comments welcome.