There are two ways to be a good forecaster, and they're different skills. One is being right a lot. The other — calibration — is having your confidence mean what it says: when you claim 70%, the thing happens about 70% of the time. Most experts are badly miscalibrated and never find out, because nobody checks.
There are two different skills hiding inside the phrase 'good forecaster,' and confusing them is how people get fooled. The first is being right a lot. The second — calibration — is subtler and rarer: it means your stated confidence actually corresponds to reality, so that when you say 70%, the thing happens about seventy percent of the time. Most experts are badly miscalibrated and never discover it, because almost nobody keeps the score that would reveal it. This essay is about that second skill, why it's the one I care about most, and how a single chart exposes whether anyone has it.
Start with why accuracy and calibration aren't the same. You can post a glittering hit rate and still be miscalibrated — just predict heavy favorites and avoid the hard questions. And you can be beautifully calibrated without being a genius; you only have to be honest about how much you don't know. Calibration is not cleverness. It's honesty about uncertainty, made measurable — which is exactly why so few people who sell forecasts will show you theirs.
Here's the precise definition, because the precision is the point. A forecaster is well-calibrated when, across all the occasions they assigned a given probability, the event occurred that fraction of the time. Gather every call where I said roughly 70%; if close to 70% of them happened, that confidence bucket is calibrated. Do it for 30%, for 90%, for every level. Calibration isn't a vibe — it's an arithmetic property of your whole record.