The market for predictive analytics, software to forecast the future, is projected to grow on the order of 25 to 30 percent per year over the next five years. That market is dominated by packages advertising one variety or another of “artificial intelligence.” In the midst of this technical frenzy, rather less attention has been devoted to the role of natural intelligence, implemented in carbon rather than silicon, in forecasting. This volume is a compelling report of an important exception.
The first three chapters report the background of the research reported in the book. The initial theme, and a recurring one throughout the book, is that without concrete ways of measuring success and failure, knowledge cannot advance. In general, human forecasting has not been subject to such discipline, but Phil Tetlock, a political psychologist, has been working to reverse that trend. His 2005 study [1] reported the results of experiments with 284 experts attempting to forecast about 28,000 questions. This volume showed that, in general, people don’t forecast very well, but some people do better than others. Tetlock classified his experts as either “foxes,” people with broad, eclectic interests and no ideological commitment, and “hedgehogs,” people who view the world through a single lens, and found that foxes had a clear edge over hedgehogs in anticipating the future.
The advantage of the foxes was not strong enough to satisfy the commercial demand for forecasting, and private investment has doubled down on silicon. But chapter 4 reports interest from another quarter. Tetlock’s research, as well as recommendations he made on committees of the National Research Council, drew the attention of the US intelligence community; in 2011, its research activity, IARPA, with heavy input from Tetlock, launched a four-year program with five research teams, each recruiting a cadre of human forecasters and deploying them on forecasting problems provided by the sponsor. (Full disclosure: I led one of the other teams.) IARPA called the project ACE, for “aggregative contingent estimation,” and most of the teams approached the problem as one of crowdsourcing, how best to aggregate forecasts from multiple human forecasters to achieve the “wisdom of the crowd” effect. Based on his previous work, Tetlock’s team (the Good Judgment Project, or GJP) focused attention more on the performance of different personality types and the benefits of training intended to cultivate foxiness.
By the end of the first year of the program, the GJP had achieved performance levels that IARPA had set for the overall program. But the team found that this outstanding performance was not uniformly distributed. They identified a small group of the forecasters who greatly outperformed the others, a group they termed “superforecasters.”
Chapters 5 through 8 tease out the characteristics of these unusually successful people. Chapter 5 looks at the contribution of intelligence. Superforecasters are more intelligent than ordinary forecasters, but not much more intelligent, compared with the much larger gap between forecasters and the general public. Their main advantages came through their lack of ideological rigidity and some teachable reasoning principles, such as attention to base rates and building up estimates from components (a process Tetlock calls “Fermi-izing,” after the physicist famous for this approach). Chapter 6 reports that superforecasters have higher numeracy skills than others, and explains the benefit of this not in their ability to calculate faster, but because it reflects a probabilistic way of thinking and a respect for intrinsic uncertainty. Chapter 7 observes that superforecasters pay attention to the news and tend to update their forecasts frequently, by small amounts, integrating new information in a manner reminiscent of Bayesian updating. Chapter 8 describes their intellectual humility and openness to modifying their previous estimates. Together, these chapters offer a disciplined outline of how to avoid the biases classically described by Kahneman and Tversky [2], which is not surprising, since Tetlock is a long-time colleague of Daniel Kahneman.
Chapter 9 returns to the implication of “aggregative” in the ACE acronym. When the GJP identified persistent superforecasters and grouped them in teams to work together, their performance improved even further, showing the value of crowdsourcing--if one has the right crowd. The concepts developed here add to the growing literature on crowdsourcing [3].
The last three chapters explore possible challenges to the notion of a superforecaster. Chapter 10 observes that the characteristics of superforecasters that the GJP identified, including lack of ideological precommitment, intellectual humility, and constant willingness to reevaluate, seem at odds with the attributes of confidence, decisiveness, and commitment to vision that characterize effective leaders. Yet a good leader must also be effective in anticipating upcoming events. The chapter illustrates how these contradictions can be reconciled by refining our understanding of the characteristics in question. Chapter 11 explores how meaningful superforecasting is in view of a “black swan” theory of history that ascribes all truly meaningful change to the cascading impacts of events far out in the tails of distributions, and concludes that while the black swan view should inspire humility, modest short-term forecasting improvements are still of value. Chapter 12 faces the challenge that the empirical approach to forecasting that Tetlock has pioneered has a chance, given the entrenched interests involved in issuing forecasts that can be rationalized with any outcome, and offers a cautiously optimistic assessment.
An appendix contains the brief training document that the GJP gave its forecasters.
Phil Tetlock is a political psychologist, not a computer scientist, and his narrative is cast in behavioral terms rather than computational or even formal statistical models. However, it will be of interest to three groups in the Computing Reviews readership. Cognitive scientists will readily translate this report into models that will inform further research and lead to new techniques in artificial intelligence. Experts in human-machine interaction will want to develop techniques to help human users become more like superforecasters in their thought processes, and students of crowdsourcing and collective intelligence will be led to explore more deeply the relationship between the dynamics of the crowd and the quality of individual crowd members.
The book is Tetlock’s story, but the coauthor, Dan Gardner, deserves credit for the accessibility of the style in which these important results are presented, while detailed endnotes will satisfy those who wish to explore further. The result is a self-help guide to effective thinking and forecasting that is solidly grounded in empirical research. Now that Tetlock has clarified the facets of natural intelligence that enable superior forecasting, one hopes that researchers in machine forecasting will explore how to incorporate these characteristics into the artificial algorithms that dominate the current marketplace for predictive analytics.
More reviews about this item: Amazon, Goodreads
