The Forecasting Proficiency Test: A General Use Assessment of Forecasting Ability

Mark Himmelstein, Sophie Ma Zhu, Nikolay Petrov, Ezra Karger, Jessica Helmer, Sivan Livnat, Amory Bennett, Page Hedley, Phil Tetlock

11/18/2024·Read paper

Abstract

Accurate predictions about the future are crucial for optimizing policy choices, but finding skilled forecasters is challenging. Large forecasting tournaments can identify generalist superforecasters who make reliably accurate predictions across domains, but such tournaments take months to complete and do little to explain why these forecasters perform at a consistently high level. We solve these problems by developing a one-hour Forecasting Proficiency Test (FPT) that explains 75% of the variance in forecasters' accuracy about a wide variety of events and provides new evidence about the mechanisms that make some people better forecasters than others. Specialized cognitive abilities, including base rate sensitivity and Bayesian reasoning, uniquely predict forecasters' accuracy even when controlling for more traditional measures of general intelligence. The FPT also includes carefully chosen real-world forecasting questions that produce interpretable and exchangeable test scores. The top 20% of testees made forecasts with accuracy comparable to previously identified superforecasters; and two-thirds of superforecasters had FPT scores above the 75th percentile. The creation of an effective test of forecasting proficiency expands our understanding of human intelligence and provides an important tool for decision-makers who rely on accurate probabilistic predictions to make high-stakes decisions that impact the lives of millions.

Make this part of your paper trail

Save this paper to a shelf, write a review, and keep your own notes.

Community Reviews

No reviews yet. Be the first to review this paper!