Do multiple choice questions make us biased?

This BBC podcast is supported by ads outside the UK. It's time to see what you can accomplish with Shopify by your side. So, we can now listen to your podcast. Another sweet! I'm trying to win sweets. Aww. You chose the right one first that time, so no sweet for you, sweet for me that time. Honestly, if maths at school had been like this, I think I'd have been top set and ended up as a prize-winning mathematician. Might be a bit old, by the way, those are really props, though. A prize-winning mathematician with terrible teeth. Oh, wow, that is... Is that really rank? That's like a pear drop. Why am I putting my dental health on the line? Well, this is CrowdScience from the BBC World Service, the programme that peers under the cups of scientific mystery to uncover the sweet truths beneath, with the help of questions from our listeners, and they're never stale. I'm Alex Lathbridge, and today I'm hunting down answers to a question from Listener Griffith in Accra, Ghana. here's the thing listener griffith isn't just a crowd science listener he's also a listener to another show here at the bbc world service science headquarters maybe you've heard of it as well it's called unexpected elements and if they're known for one thing it's the halftime quiz hello i'm alice and it's time for the unexpected elements quiz this week we've been inspired by Basically, I realised that the answers to the quiz questions were quite reliably B and C and not A. So the question was, was the bias for the B and C intentional or unconscious? And then are there ways as a student to explore this bias to make better choices in exams? Have a think about that and I'll be back later with the answer. Science is a wonderful thing. It gets us to the moon and back, helps us comprehend the mysteries of the cosmos and decipher the depths of our DNA. More importantly, it can help you cheat on your exams. I remember when I was also in school, I came up against many multiple choice exams. I remember there was a joke, a running joke, that if you don't know the answer, choose B or C or D and it's almost never A and so sometimes I used to do that I don't know how successful I was but I just used to do that when I didn't know the answers. You never picked A you always went B C or D. Yeah yeah. Why? Well I guess like if I want to hide something from someone I wouldn't put it on the surface like if I was a lecturer I wouldn't want the answer to jump out at them immediately if they don't know. You know, A is the first answer that will come, and B and C and D go deeper and deeper and deeper. That's my logic behind it. Well, unfortunately for the Unexpected Elements team, they sit on the same row of desks as me. And there's no escaping the eye of scientific scrutiny. So I grabbed the first two Unexpected Elements producers I could find to answer for it. Hi, my name's Lucy. I'm one of the producers on Unexpected Elements. I've been working on it for about six months. And my name's Alice. I'm also one of the producers on Unexpected Elements. And I've worked on it for about a year and a half now. What makes a good quiz question? It's got to be surprising. OK. Yeah, I feel it can't be too hard either. You can't do something so obscure that no one's got any chance of getting it right. And also at least one decent science pun, I think, per quiz. Yeah, I think puns are essential. So our listener Griffith, he's noticed the answer to the quiz, it's hardly ever A. It's always B or C. And so the real question I have is, what have you got to say for yourselves? Is this corruption at the heart of unexpected elements? It's anti-A bias. I think it's an acceptable form of corruption. The way I write the questions, it just ends up being C. Like say it was like, how tall is like this mountain? I want the answer to be the most surprising. so it's generally going to be the biggest one. It just naturally ends up being C. And I've tried to write it differently, but it just feels wrong. Whereas Lucy tends to go for C, I'd like to go for B, so I'd pick something nice in the middle that might be the more logical answer. But I have to say, when I realised I was going for B more often, I then thought, oh, I'm going to throw in some A answers now because I've just been biased towards B. And I'd say I barely ever go for C, which is the one that Lucy is a seagull. So, you know, I've been trying to throw in some A's and sometimes it does feel just sort of impossible. Do you think then that making option A be the answer is somehow too obvious? Yeah, and I think if you got the answer straight away, if you knew it, you're going to be like, oh, yeah, that's it. And then you might turn off the rest of the quiz. Yeah, we want people to listen to the whole quiz. I don't want people to switch off after option A because I'm like, oh, I've got it now. I want them to listen to my voice for the whole minute. I put the effort in. It is a lovely voice as well. So, you know. Are the unexpected elements producers unique? Or can we all fall foul of this pattern picking behaviour in our day-to-day lives? Well, this is the point where I needed to enlist the help of an actual mathematician. Hi, I'm Kit Yates. I'm a professor of mathematical biology and public engagement here at the University of Bath. So our listener Griffith wrote in because he spotted that the answer to the unexpected elements quiz is always B or C, but hardly ever A. Now, you're a mathematician. What's going on? So I think if you were genuinely choosing these answers, placing them at random, sometimes it would be A, sometimes it would be B, sometimes it would be C. But we tend to exhibit a number of biases, which mean we're not very good at being random. It might be that we're being influenced by something called middle bias, which is our propensity to choose the middlemost option when we've got a bunch of options to choose from. So you see middle bias in SAT tests where students get a choice between four options, A, B, C and D. And if they genuinely don't know the answer, they tend to choose B and C with a higher propensity than A and D. You also see it in sort of games like battleships. People tend to put their battleships disproportionately towards the middle and away from the edges. You even see it in people's choices of toilet cubicles, apparently. The thing is, this middle bias might skew our instincts in multiple choice questions, but it actually exists for a good reason. Kit says that, in normal life, if you don't know something, it actually a great tactic to guess towards the middle of your options For example if I had to guess a stranger height with no other information it would make sense for me to guess somewhere around the average human height. Some things are distributed according to what's called the normal distribution or the bell curve. So-called because when you map it onto a graph, it does kind of look like a bell, small on one end, large in the middle, and small at the other end. Men's heights or women's heights, for example are distributed according to this bell curve where genuinely more of them do sit towards the middle than towards the outer edges so it's an okay assumption in those cases but when we're trying to genuinely uniformly so equally choose between different options that's when we really fall foul of middle bias so why are we susceptible to fallacies like these i think the experiences that we have had in the past both in our lives but also evolutionarily have primed us for spotting patterns and we think we see patterns in even random data this is this phenomena called pareidolia where we we seem to spot a pattern in in a random stimulus like um seeing jesus in a piece of burnt toast back in the day when we were out hunting or gathering we're out in the woods and we see a sort of stripy pattern in the trees it could have been just the sun shining through the leaves or it could have been maybe a tiger it was important for us to be able to spot those patterns but sometimes we get it wrong when we interpret what those patterns mean the brain's amazing ability to spot patterns amongst the noise has come at a cost we struggle to deal with data that is patternless true randomness that's why the unexpected elements producers are so reluctant to make the right answer a a just doesn't feel very random like the vibes are off and this is true for everyone even the really clever people who understand maths like kit they are biased in favor of answers that feel right i'll give you an example the birthday problem so the birthday problem basically says how many people do you need to have in a room before it becomes more likely than not that two people in that room will share a birthday so the probability of two people sharing birthday goes above 50 okay 365 days in a year let's say you need 182 that is the answer that most people would give right so you divide 365 by two and you get to about 180 something but actually the answer is surprisingly small it's just 23 how it's crazy isn't it it's totally mad So by the time you've got 23 people in the room, there are 253 different pairs of people. Assuming all those pairs are independent... This is the point where things became exponentially more complicated and I completely zoned out. ...to the power of 253 or multiply it by itself 252 at a time. The point is, with 23 people in a room, the chance is just over 50% that two people will share a birthday. Even though that's true, it just feels wrong, you know? If you want proof of that, you can go to the World Cup, right? In the World Cup, they have squads of 23 players. And if you so desire, as I do, to go on to Wikipedia and to download all the birthdays of all the squads of all the teams that play, you can then match up and see how many of those squads of 23 players have two players that share a birthday. And actually, in the Men's World Cup in 2014, it was exactly half of them. It was 16 out of the 32 teams. So you see it in the real world. So, taking it back to multiple choice questions. One of the most famous examples of the way that our instincts about probability can let us down is in a problem devised in the 1970s, known as the Monty Hall problem. It comes from a game show which was called Let's Make a Deal, and the host of that game show was a guy called Monty Hall. There were three doors that you had to choose, numbered one, two and three. and behind one of the doors was a car and behind two of the doors was a goat so you're potentially going to end up with a goat two times out of three but you might win this car okay the way it worked was you got to choose a door and then the host monty he would open another door which had a goat behind it and then you would be left with these two doors one with the goat behind it one with the car behind it and you had to choose whether you wanted to stick with your original choice or whether we wanted to switch and go with the other alternative. What should you do? Should you stick or should you twist? What do you think? It's a 50-50 chance at that point. Yeah, that's the intuition, but it's actually not a 50-50 chance. Actually, if you switch two times out of three, you're going to win the prize, and only one time out of three are you going to end up getting the goat. It feels like a 50-50 chance, but it isn't. So I'm sat in Kit's office completely baffled. Fortunately, Kit came prepared. Tell you what, let's play the game. And because mathematicians hate small sample sizes, we're not going to play it once. We're going to play it ten times. We've got ten sets of three cups, numbered one, two and three. So under one of the cups is a sweet and under the other two cups there's nothing. OK, let's begin. So I'm going to start with cup number one. Okay, then I am going to show you cup number two. I pick a cup, but I don't look under it. Then Kit removes one of the empty cups from the game. And then I switch my choice to the other remaining cup. So you switch. All right, I'm going to pick number three. According to the maths, this should be a solid suite winning strategy. There's a suite underneath. Very good, next round then. I win a suite, then another. I win four suites in a row and I begin to realise that I am a genius. Another suite! Oh, smashing it. Take that statistics. Right, next one that you're going to choose. Then my luck turns. Oh. So that time you picked the right one first, so you didn't end up winning when you switched. In total, using Kit's switching tactic, I win the game six times out of ten. So it's close to what we'd expect, which is 66%, right? Two thirds of the time that you would expect to win. Okay. So basically, because there's more wrong options, there's only one correct option. There are two wrong options. So 66% chance you'll pick a wrong option. And then you switch. You always get the right one. Yeah. And that look on your face when it was like, oh, it's about what you choose first. And then you swap, you always get the opposite. That's it. That's the magic of understanding the probability. It's a beautiful thing when it comes. yeah yeah it's great so our pattern spotting brains have given us these tricky cognitive biases which skew our decision making but if you think that multiple choice questions are the only way that these biases trip us up you're very much mistaken more brain boggling examples for you coming up next starting a business can be overwhelming you're juggling multiple roles designer, marketer, logistics manager, all while bringing your vision to life. Shopify helps millions of business sell online. Build fast with templates and AI descriptions and photos, inventory and shipping. Sign up for your one euro per month trial and start selling today at Shopify.nl. That's Shopify.nl. It's time to see what you can accomplish with Shopify by your side. I understand that you want to listen to your podcast, so I'll keep it short. Because if you think it's important to make a duurzaam keuze, can ASR maybe help? Well, I think, how then? Well, for example, when it's a lot of money that you love are at Schade. Will you know more about the insurance where a duurzaam schadehersetel can be? Go to asr slash duurzamekeuzes This is ASR for you and a duurzame community ASR does it So we can now listen to your podcast thanks to a question from listener Griffith in Ghana. They can really trip you up in unexpected ways, particularly when we're playing games of chance. So my husband and I were in Las Vegas at a very low-end, inexpensive casino. We were playing a little bit and we were watching people playing. This is Rachel Croson, queen of the roulette table, master gambler of the Las Vegas Strip and professor of economics at the University of Minnesota. I do research in experimental and behavioural economics, how people make economic decisions, what mistakes they might make and how you can help them make better decisions. Speaking of bad economic decisions, back to Vegas and the 10 cent roulette wheel. There was one guy there who had a piece of paper and he was very carefully keeping track of all the numbers that had come up. And he had data for hundreds and hundreds of spins, so he must have been there for hours. If you're not familiar with roulette, let me paint you a picture of what's going on here. A roulette wheel is a sort of spinning disc laid flat on a table, and there are little slots all around the edge of it. Each slot is numbered from 0, or sometimes double 0, through to 36. And to play, the dealer spins the disc around, and then throws a tiny ball into the middle, where it bounces, almost chaotically for a while, and then eventually comes to rest in one of the little slots. You can bet which of the numbered slots that you think the ball is going to land in, and if it lands in your slot, you win. Of course, with so many numbers on the wheel, it doesn't land on your one very often, probably around one in every 38 times, or at least that's how it should work. So we look at his data, and there is indeed one number, I think it was 18, that had come up way more times than you would expect. And so we talked to him and we said, so what's going on with number 18? And it's possible that you would actually have a biased wheel in a very low level casino. Of course, as the wheel was biased in favor of 18, you should be betting 18. But that's not what this man was doing. He said, it's come up so often, it's done. And he said, I'm not going to bet 18 anymore. So this was so amusing and frustrating to us because he had collected all the right data and he had come to exactly the wrong conclusion. That number is done. It's over. It's never going to come up again. What was going on in his head? Yeah, that's a great question. Of course, it's always hard to tell what's going on in any one individual person's head. And so that's really what led me to the gambler's fallacy. The gambler's fallacy, just like middle bias, is one of those tricky little cognitive biases that trip us up when maths doesn't work the way that we think it should. Say you and I are playing a game, flipping a coin. Heads, you win. Tails, I win. Let's say it comes up tails twice, then three times, then four times in a row. If you find yourself thinking, oh, four tails in a row, it's got to be heads next. Then I'm afraid that you just fell victim to the gambler's fallacy, because the truth is the chance is still exactly 50-50. The wheel has no memory of what came before. So the previous spins, the previous outcomes have no predictive value about what's going to happen next time. And the gambler's fallacy, in contrast, says I believe that once something has come up in the past, it is less likely to come up in the future. Rachel also studies another common fallacy, which pulls us in the other direction, called the hot hand fallacy. Let's say that we flip the coin over again, and this time it comes up heads. Again, let's say four times in a row. Now you're winning, over and over and over. If you find yourself thinking, oh, I'm on a winning streak, it's got to be heads again for sure. I'm on a roll, I'm on a streak, I can't lose, right? Everything I do wins. I'm afraid that this time you've fallen victim to the hot hand fallacy. So the gambler's fallacy is a biased belief about the outcome of a gamble. The wheel's going to come up red, the wheel's going to come up black. The hot hand, in contrast, is a biased belief about whether I'm going to win or lose. It's a belief about me. So if I'm hot, whether I'm betting on red or black, I'm going to win. It doesn't matter. When it comes to researching the gambler's fallacy and the hot hand fallacy, are you able to do it at the university or are you having to go down to Vegas? So yes and yes. You do a lot of it at the university. You do it with students in labs, you do it with coin flips and dice rolls. But at some point, you're going to want to look at people making real significant gambling decisions in the context that they would normally make those decisions in. So when we did our experiments with the Hot Hand Gambling Fallacy and Roulette, we originally had planned to send a PhD student to go and stand behind the roulette table and observe all the bets people made and write them down. And we were all set to do this. And then the individual at the casino we were working with said, oh, no, no, don't do that. We'll just send you the videotapes. And so our poor PhD student who had been hoping to go to Las Vegas and spend about a week collecting data instead was in the basement in Philadelphia watching videotapes frame by frame and coding up everybody's bets. So in that study, what were you doing, apart from making this boy very, very sad? So we were looking for how betting patterns changed when particular outcomes happened one time, two times, three times, four times, more than five times in a row. And we saw whether people changed their bet to the other one. So that's kind of a very clean test of the gambler's fallacy, right? If after reds come up four times, you switch to black or you stop betting red, that means you think red's less likely. Then the second thing we did for the hot hand is look at how people's bets changed after they won versus after they lost. And if after they won, they increased their bet, that was a signal that they thought they were hot, they're betting more. And what did you find there? So generally speaking, those beliefs are correlated with each other, but that correlation is not 100%. And they are absolutely people who believe in both the hot hand and the Gamos fallacy. So they're related, but they're not identical. And do we see it in sort of real world scenarios outside of the casino? Absolutely. I also teach a class on behavioral finance. And one of the regularities in how people deal with their investment portfolios is called the disposition effect. And the disposition effect means that people tend to sell their winners, their stocks that have appreciated, and hold on to their losers. And this is exactly what you would expect if people had a gambler's fallacy belief. Because this stock has gone up, it's due to go down, and so I should sell it. Or the stock has been going down, it's due to go up, and I should hold on to it. Why do these fallacies happen? Is it something intrinsically human in our brains? Yeah, that's a great question. There's been a lot of speculation about that. Certainly, there's a body of research that suggests that while roulette wheels and coin flips are indeed autocorrelated, the probability of what happens next is independent of what happened before, most of the world probably is autocorrelated. If you a hunter and you looking for where the wheat is the most likely place that the wheat is going to be growing is where the wheat grew last year right So you can absolutely imagine situations where having biased beliefs could be a positive rather than a negative. So there are all sorts of cognitive biases that can trip us up in games of chance and probability. But of course, some of the best games are less about chance and more about skill. So what kind of cognitive biases might we be vulnerable to when playing a game that mixes chance and skill? Poker is a game of skill with a chance element. In poker you can, and this is the absolutely crucial thing, in poker you can win while holding the worst hand. You can outplay other people and win. Now this is someone who knows a thing or two about games of skill. My name is Maria Konnikova. I am a writer, a psychologist. I worked on self-control and risky decision making. And I also happen to be a poker player. How did this come about? How did you first come to poker? That is a wonderful question, Alex, because I did not grow up playing games at all. I had no idea what poker was. And I actually got into it from the perspective of game theory. So after my second book came out, which was The Confidence Game, the book was doing incredibly well. I was on book door. Everything was great. And then I got very, very sick and nobody could diagnose what was wrong with me. And just everything started going wrong. all at once. You know, my mom lost her job, my grandmother died, my husband lost his job. And I realized that I wanted to write about the role that chance plays in life. And one of the books that was recommended to me was John von Neumann's Theory of Games. And I was absolutely floored to find out that one of the geniuses of the 20th century was a poker player and that von Neumann believed that solving poker would basically solve the most complex strategic decision making in the world. And so I basically thought, okay, you know, what is this poker thing? Why don't I learn this game from zero, literally? When I started, I did not know how many cards were in a deck. And why don't I use poker kind of as a metaphor for life, as a way of looking at decision making in a very practical way. What started as a thought experiment? Well, let's just say it kind of took off. These days, Maria is a pretty significant figure in the poker world, although she wouldn't describe herself that way. I have won one world series of poker bracelet and I've won some pretty big titles. But honestly, I am a mediocre poker player who's just gotten very lucky. When you came to POCA, you came as a complete novice. Yes. Well, background in psychology and really understanding the mind. So what were some of the cognitive biases that you had to contend with? You know, did your instincts lead you astray in any way? Oh, I mean, I think that you have to contend with all of your cognitive biases all the time. I mean, it doesn't matter how good of a psychologist you are, you're still going to exhibit them. I came with all of them. I mean, you name it, I had it. A sunk cost fallacy, hot hand fallacy, gambler's fallacy, risk aversion, loss aversion. And poker is a kind of laboratory environment where you can clean up some of those biases because there is money on the line. When you act in a biased way, you lose. and you have to learn how to make a decision knowing that you have these biases. Maria says that one of the biggest biases our brains trip us up with in poker is known as attribution bias. When we're doing well in life or in games, we tend to attribute it to our skill. I did good because I am built different. Therefore, I deserve it. But when we're losing, we attribute it to plain old bad luck. I did bad because fate was against me and I didn't deserve it. I think one of the single most important things that poker can teach anyone is that process and outcome are not the same thing. So process is the skill part. It's your decision making in any given point in time. And then there's the outcome, which is what happens. That's the chance part of the equation. And that's something that is now outside of your control. We tend to conflate the two because it's just so incredibly easy to do. And what poker teaches you to do is to disambiguate them, to tear them apart and to actually ask, OK, what was my process? Did I make the right decision? That's all that I can do because that's what I control. In poker and in life, there's no such thing as 100% certainty. It just does not exist. I can always make perfect decisions and things will still go wrong. and I can be an idiot and things go right. I mean, that's a perfect way for me to talk about my life. Yeah, I think that poker is the perfect analogy for life. Life is a game of incomplete information. There are things that you know, things that I know, things we all know. And our goal in life is to make good decisions, live as good a life as possible. No matter what we do, there's going to be uncertainty. You know, in some ways it's funny because poker is such a competitive game, but it's made me so much more zen because I've learned that once I've done everything I can, I just need to let go and whatever happens, happens. Well, would you look at that? A show about multiple choice quizzes ends up being rather philosophical. I like it. It turns out that none of us can escape the cognitive biases built into our brains, but those biases aren't flaws. Our instincts for spotting patterns has helped us survive and make decisions in a complex uncertain world. We may not be able to control the hand that we're dealt, but if we can let go of the illusion of control, we might be able to play it a little better. In the meantime, Griffith, on the next Unexpected Elements quiz, I'd go for option A. Really, it hasn't come up for ages, so it must be due now. I mean, don't put any money on it. Thanks so much for a great question, and please, take it away with the credits. That's all from this episode of CrowdScience from the BBC World Service. This week's question was from me, Griffith, from Accra in Ghana. The show was presented by Alex Lathbridge and the producer was Emily Knight. If you have a question on any science subjects and you want the team to investigate, why not email crowdscience at bbc.co.uk. Goodbye. Starting a business can be overwhelming. You're juggling multiple roles, designer, marketer, logistics manager, all while bringing your vision to life. Shopify helps millions of business sell online. Build fast with templates and AI descriptions and photos, inventory and shipping. Sign up for your one euro per month trial and start selling today at shopify.nl. That's shopify.nl. It's time to see what you can accomplish with Shopify by your side. To be continued... Come across your mother and table.