BBC Sounds Music Radio Podcasts Hello and thanks for downloading the More or Less podcast, with a programme that looks at the numbers in the news, in life and in highly ambiguous language. I'm Charlotte MacDonald. When you read in the news, you'll often stumble across words that are meant to communicate the probability of something happening. A terrorist attack is a realistic possibility, the spread of a certain strain of viruses, highly likely. The relegation of your favourite football team is possible, even if you did get a last minute equaliser against Liverpool. But when you hear these terms, do you really know what kind of probabilities they're trying to get across? Do you know how likely, likely is? And what probability, probable, is meant to convey? In some cases, it seems you probably don't. I'm Adam Kaczarski. I'm a professor of epidemiology and author of Proof the Uncertain Science of Certainty. Adam has a very healthy obsession with the problem of communicating probabilities, and it would be remiss of us if we didn't start out by making sure you understand what the numbers you're going to hear actually mean. So what does it mean if we say there's an 80% chance of something happening? Intuitively, we can think of the chance of something happening as the proportion of those events that would come to fruition in reality. So, for example, if there's 10 events, all of which that have an 80% chance of happening in the next year, say, on average, we'd expect eight of those 10 to happen. If it's a 10% chance, we'd only expect, on average, one of those 10 events to actually occur. OK. So we can express probabilities with numbers, and we can also express them with words like probable, unlikely, and so on. Ideally, the words and numbers match up somehow. But that's not always the case. A real standout example for this kind of problem came in 1951 when a CIA analyst called Sherman Kent wrote a report that said that there was a serious possibility that what was then USSR would try and invade Yugoslavia that year. A serious possibility of war seems like quite an important thing to be clear about. He then bumped into someone who was a chairman of an important committee in a corridor and the chairman said, what did you mean by a serious possibility? And Kent said he thought about 65%. And the chairman said, I thought he meant something much lower than that. And Kent kind of had this panic and went back and talked to a lot of his team who had been writing these dozens of reports and realized that even in the team who are co-writing these things, they had very different numbers in mind for the kind of language they were using. The invasion never happened. But this ambiguity between language and numbers rumbled on over the decades. However, Adam says that in the last 25 years, there's been a real effort to pin them down, particularly from the intelligence agencies which are trying to communicate serious risks to politicians. So there was real effort to introduce in the UK what was known as a probability yardstick or saying if you're using these kinds of words, this is the probability that you mean. So if you're using words like highly likely, you're saying that there's an 80 to 90% chance that that's going to occur. If you're using the phrase realistic possibility, you mean a 40 to 50% chance. James Bond and his analysts in the back office should be communicating clearly now. But what about the rest of us? This probability language isn't just found in government reports. It also percolates out into the public in new stories about wars and pandemics and climate change. Does the public have a clear idea what these words actually mean? Adam says some small studies have looked at this, but he wanted much more data. So he built a quiz. There were two main parts to quiz. The first was comparing phrases. So you had a series of pairs, and it might say for each pair, which has the higher probability of something occurring. The second part of the quiz got people to put specific numbers to phrases. So if you're given a phrase like may happen, pick a number between one and 100% that you think matches the probability of that phrase occurring. More than 5000 people took part in Adam's probability quiz. You can have a go yourself if you click on the link on our website. And this gave Adam a pretty good idea of which words were highly ambiguous and which everyone agreed on. One of the things that was possible was to see where the highest amount of ambiguity was. So in terms of just the variation in what people say, the most consistent was the phrase about even, which you'd hope that perhaps people would land on about 50%. There was also more clarity in some high and low probability phrases. For instance, most people thought the phrase will happen meant there was a 100% chance and almost certain something around 95%. At the other end, most people thought the phrase almost no chance meant between a one and five percent chance. The phrases remote chance and highly unlikely both came in between one and 10%. Those are the phrases where you can be pretty sure people know what you're saying. However, some were far less clear. The most variable phrase was realistic possibility. So one that actually happens to be in intelligence guidelines and often used internationally in headlines to report risks of events. Respondents to Adam's quiz thought the phrase realistic possibility could mean anything from a 10% chance of it happening to being a near certainty with the average answer just over 50%. So it really does seem like a word that means a lot of different things to a lot of different people. The terms probable and likely both had ranges from around 50 to 100% for the averages just under 75%. Adam also asked people to compare probability words side by side to see if people were using probability in a consistent way. So here goes. Let's try a little quiz of our own. Which is more likely to happen? Something that is very unlikely or something that is highly probable? Okay, that one was easy. Obviously, highly probable is the answer. How about this though? Which is more likely? Something that might happen or something that could happen? Turns out people's judgments on these close comparisons are not consistent at all. Some phrases particularly that might happen, it could happen, which arguably are just very hard to pin down. But even if you give people that and then give them the same judgment a few questions later, they'll flip between them somewhat arbitrarily. While the study was self-selecting, he did end up getting a reasonable spread of responses from people in different groups. There was a bit of a trend, for example, with values in education. So typically, people who had more degrees tended to be a bit more pessimistic. We also found that people responding from the US were slightly more optimistic in their answers to all things being equal compared to the UK. These, however, were only small effects in the scheme of things, moving the numbers by a few percent. Really, the big picture here is that some probability language is just very ambiguous. That's a problem if you're trying to warn people about serious risks. But what should we do about that? For Adam, one thing is just making sure that the people communicating risk, whether they're scientists, politicians or journalists, know the risk. Just ensuring that what the person coming up with an estimate is saying is what the person is hearing at the other end. The second is to push the people calculating risks, not to hide behind these words to avoid accountability. All of us, I think, can have some benefit in actually challenging ourselves to make a judgment that we can really go back and ask if we genuinely got that right, rather than having these kind of ambiguous words where we can always wriggle out of it later if we want. Thanks to Adam Kaczarski. If you've seen a number in the news you think we should take a look at, email more or less at bbc.co.uk. We will be back next week. Until then, goodbye.
