judgement
Calibration - grade your maybes
The rule
Keep score of your guesses. Do the things you call "70% likely" really happen about 70 times in 100?
Where it flips
A few calls tell you very little. Luck can decide a small number. The fix is to check your hit-rate across many guesses over time, not a handful.
Keeping score is more than just right or wrong. A good guesser checks something called calibration. You gather all the calls you made at "70%" and see if about 70 in 100 came true. If your "90%" calls happen only half the time, you are too confident. Now you know, and you can fix it. This honest mirror turns guessing from mere opinion into a real skill you can improve.
A worked example
Over a year Aayra makes fifty "80% likely" calls. Only 55% happened. That is a clear sign she is too confident, so she widens her ranges. Without checking her hit-rate she would never have known. [illustrative]
How to spot it
- ·calls grouped by stated confidence
- ·stated odds compared to real hit-rates
- ·overconfidence measured and corrected
Philip Tetlock & Dan Gardner · Superforecasting