Understanding probabilities

“70 %” does not mean “this will happen”. It means: if this situation could be replayed ten times, you would expect the event to happen seven times and not happen three times.

Those three times in ten are not a failure of the model. They are part of what the model announced. A forecaster who says 70 % and is always right is not telling the truth about their uncertainty: they should have said 100 %.

A single forecast cannot be judged. An event called at 20 % that happens proves nothing at all: 20 % is one time in five, and one time in five happens often.

What can be judged is a set. Take a hundred forecasts called at 70 %: if roughly seventy come true, the forecaster is well calibrated. If ninety come true, they are too cautious and should have said more. If fifty come true, they are too sure of themselves.

That is the only honest measure, and it takes time and volume. Conclude nothing from one forecast; conclude from many.

Predictly shows two things that look alike and do not measure the same thing. This is the commonest misreading.

Probability

How likely the event is to happen.

Confidence

How far the answer rests on evidence that is plentiful, independent and serious.

All four combinations exist, and each reads differently:

• High probability, high confidence: the outcome is clear and the dossier is solid.
• High probability, low confidence: the outcome looks clear, but on very little. Worth revisiting as soon as anything comes out.
• Low probability, high confidence: it is well established that this is unlikely. That is useful information, not a contradiction.
• Low probability, low confidence: little is known, and the little that is known does not argue for it.

Predictly rounds to the nearest 5: 55 %, 60 %, 65 %. Never 61.4 %.

A figure to the decimal implies a measurement nobody made. On a match, a game launch or a vote, the difference between 61 % and 62 % does not exist in the data: it exists only in the arithmetic that produced it. Rounding is a way of not suggesting a precision that is not there.

Before looking at what is being reported about a subject, a useful question is: how often does this kind of thing usually happen? That is the base rate, and it is the thing most often forgotten.

Delays are common in video games; transfers described as imminent do not always go through; an incumbent stands again more often than they lose. Dramatic news moves the estimate, but the estimate starts somewhere, and that starting point matters as much as the latest story.

The word next to the percentage depends on the percentage itself and on the gap to the next outcome. It is fixed per band, so the same number always reads the same way.

Likely

65 % or more.

Favourite

The leading outcome, clearly ahead of the next, without reaching 65 %.

Open

The leading outcome, but fewer than 10 points ahead of the next. In other words, nothing is settled.

Possible

35 % or more, without leading.

Unlikely

Below that.

Sometimes no percentage appears and Predictly simply says it has not found enough reliable evidence. That is not a fault: it is the evidence gate described in the methodology.

A subject that is obscure, too recent, or where everyone is relaying one single source gives nothing to estimate from. Producing a number there would be dressing an impression up as a measurement.

For the calculation behind the percentage, see the methodology. For how to ask a question, see how it works.