Nineteen people have taken the quiz.
Out of the 190 individual answers received, 97 were correct -- slightly over half. The distribution of scores on the quiz is as follows:
| Score | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
| Number of people | 1 | 4 | 3 | 4 | 2 | 3 | 1 | 0 | 1 |
In short, the respondents as a group confirm Ayres' claim that "almost everyone who answers these questions has the opposite problem of overconfidence -- they can't help themselves from reporting ranges that are too small." Ayres cites a book by J. Edward Russo and Paul J. H. Schoemaker, Decision Traps: Ten Barriers to Brilliant Decision-Making and How to Overcome Them, which I haven't read; supposedly "most" people get between three and six questions right. I'm actually soewhat surprised that you as a group don't seem all that different from the general population.
I have some other comments -- which questions seem particularly difficult or easy, what we might say about confidence intervals other than 90 percent -- but I'm hoping more people might answer, so I'll wait for that. (Although if the remaining answers are suspiciously better-calibrated that the answers so far, that might turn out to be not such a good idea.)
4 comments:
Writing as the proud owner of the leftmost datapoint... that overconfidence thing was wild. For a bunch of answers, had I opened up my range just a little bit more...
The center's of my ranges were ok, but 90% confidence? I missed that completely.
By the way, books in the old testament? How many?
Jonathan
Now I'm actually going to have to go back and check my answers. I remember think on a couple of the answers that I was picking a pretty tight range, but that I also had an obligation to pick my 90% confidence.
For a couple of them, I noticed that I thought I was either very close, or a million [virtual] miles away, and that bimodality of feeling was hard to adjust to.
For the most part, we readers probably don't know a whole lot about many of the things in questions. We are as naive as the general population.
David,
I wasn't saying that my readers would be more knowledgeable about those particular questions -- rather that they'd be more knowledgeable about probability, and in particular what 90% confidence means.
Right-oh on the more knowledgeable about probability. I think, though, that we are accustomed through practice to envision our distributions as being very central.
When we understand very little about something, our variance is higher [variance being a proxy for accuracy; our accuracy is lower]. When our SD's get big relative to the value of our mean, our distros a going to be flatter. My ego takes a hit when I can't narrow my estimates, even knowing what I know about probability.
Maybe it's that we want to externalize the variance as a feature of the possibilities, and not a characterization of our knowledge. So, I see the wide spread, and think to myself, "That can't be right," because I don't encounter many processes like it. By the by, it's an interesting anthropological problem.
Post a Comment