September 27, 2012
Human beings have to make plans and strategize for the future. As the pace of our lives becomes faster and faster, we have to do so more often and more quickly. But are our predictions any good? Is there hope for improvement?
In The Signal and the Noise, Nate Silver examines the world of prediction, investigating how we can distinguish a true signal from a universe of noisy, ever-increasing data. Many predictions fail, often at great cost to society, because most of us have a poor understanding of probability and uncertainty. We are wired to detect the signal, and we mistake more confident predictions for more accurate ones. But overconfidence is often the reason for failure. If our appreciation of uncertainty improves, our predictions can get better too. This is the prediction paradox: the more humility we have about our ability to make predictions – and the more we are willing to learn from our mistakes – the more we can turn information into knowledge and data into foresight.
Silver examines both successes and failures to determine what more accurate forecasters have in common. In keeping with his own aim to seek truth from data, Silver visits innovative forecasters in a range of areas, from hurricanes to baseball, from the poker table to the stock market, from Capitol Hill to the MBA. Even when their innovations are modest, we can learn from their methods. How can we train ourselves to think probabilistically, as they do? How can the insights of an eighteenth-century Englishman unlock the twenty-first-century challenges of global warming and terrorism? How can being smarter about the future help us make better decisions in the present?
Sometimes, it is not so much how good a prediction is in an absolute sense that matters but how good it is relative to the competition. Competition is essential to making forecasts better – but it can also make them worse if forecasters compete with the wrong goals in mind. The more accurate forecasters recognize that prediction is still a very rudimentary – and risky – science. They are motivated by truth rather than by politics, and they notice a thousand little details that bring them closer to it. Because of these attitudes, they can distinguish the signal from the noise.
With everything from the health of the global economy to our ability to fight disease dependent on the quality of our predictions, Nate Silver’s insights are an essential read.
Nate Silver is a statistician, writer, and founder of the New York Times political blog FiveThirtyEight.com. Silver also developed PECOTA, a system for forecasting baseball performance that was bought by Baseball Prospectus. He was named one of the world’s 100 Most Influential People by Time magazine. He lives in Brooklyn, New York.