A prediction-making quiz

I just read Ian Ayres' book Super Crunchers, which talks about how the large amounts of data that are now routinely collected enable better predictions than before. Sort of like Freakonomics but a bit more statistical. (Although all the math is hidden -- but I knew that going in.)

Now, there was a recent article The End of Theory which predicts that we don't need theories, we can just mine our data for correlations; I don't believe this. And Ayres talks about how some predictive models need human input -- for example, a model for predicting how Supreme Court justices will vote needs people to read previous input on the cases in order to decide whether the ruling being appealed was liberal or conservative, and also to determine what the major issues involved in the case are. But he ponts out that people are bad at predicting things because we are overconfident about our predictions.

This piqued my curiosity. Here's a quiz; I want to see how good you are at calibrating your own predictions. (This is taken from Ayres' book, p. 113.) For each of the following ten questions, give a range that you are 90 percent confident contains the correct answer. Ayres' test implicitly uses English units, but if you want to use metric (which I suspect a lot of you are more comfortable in) that's fine; I'll convert.

So, for example, if one of the questions were "What is the population of Philadelphia?", and you gave the numbers "1.2 million, 1.6 million", that would indicate that you believe with probability 90 percent that the population of Philadelphia is in that interval. (The 2006 Census estimate for this, by the way, is 1,448,394.)

Your goal is to get exactly nine of these right. Yes, I know that sounds weird! But the point is that if you get all ten right, you're proabably underestimating your own abilities to predict things. If you get eight or less, you're probably overestimating them.

Send your answers to me at izzycat AT gmail DOT com; don't leave them in comments.

Here are the questions:
1. How old was Martin Luther King, Jr. at death?
2. What is the length of the Nile River?
3. How many countries belong to OPEC?
4. How many books are there in the Old Testament?
5. What is the diameter of the moon?
6. What is the weight of an empty Boeing 747-400?
7. In what year was Mozart born?
8. What is the gestation period of an Asian elephant?
9. What is the air distance from London to Tokyo?
10. What is the depth of the deepest known point in the ocean?


Also:
1. feel free to forward this quiz to other people. (I encourage it, although there's a non-negligible chance I might regret this if I get too many answers. I'll survive.)
2. if you have stories about how you made your guess, send them to me; I may use them in a future post.
I'm not going to post the answers; none of them are hard to find. Once answers stop coming in I'll make a post about how good you are at making these predictions.

Punitive damages

From Overcoming Bias: Unbounded Scales, Huge Jury Awards, & Futurism.

Apparently jurors are able to consistently rank-order the amount of money that they think should be given as damages in various cases -- they can say "I think person A deserves more money than person B", and other people will agree with them. And it seems implied in the post (although it's never actually stated) that they're fairly consistent in things like "I think person A deserves twice as much money as person B". But their assignments of dollar amounts to people are all over the map.

(If you're thinking "jurors always give ridiculous amounts in punitive damages!", that might be because the media seems to like the really out-there cases, like the guy who sued his dry cleaners for $54 million because they ruined his pants. He didn't get the money. I am not a lawyer, but I suspect that the amounts given in damages in most cases are more reasonable-sounding, if only because the people on juries are people like you and me.)

Similar things might hold for predictions of the future -- people are better at guessing the order in which the future will unfold than the speed at which it will unfold.

The spinning woman, part two

Remember the spinning woman that I wrote aboutI wrote about last week?

Supposedly, people who saw her spinning clockwise are more "right-brained" (in the pop-science sense of creative, able to see the big picture, and so on) and people who saw her spinning counterclockwise are more "left-brained".

According to the good folks at Freakonomics, if you use college major as a proxy for pop-culture brain-sidedness, it actually works the other way -- "left-brained" people are more likely to see her spinning clockwise. (Of course, this isn't scientific -- it's just Freakonomics readers, and the sample sizes are small.)

Steven Levitt writes:

I often joke about how the information provided by someone who is incredibly terrible at predicting the future (i.e., they always get things wrong) is just as valuable as what you get from someone who is good at predicting the future. I used this strategy with some success by betting the opposite of my father whenever he’d bet a large sum of money on a football team that was sure to cover the spread.

That's definitely true. I've heard that another such "anti-predictor" is Punxsutawney Phil, the "official" groundhog of Groundhog Day. He gets yanked out of hibernation on February 2, and if he sees his shadow we're supposed to have "six more weeks of winter". (This is part of the utterly bizarre American tradition of Groundhog Day.) I read once that he's wrong something like 80% of the time, although I don't have a source for this. The Wikipedia article says that the actual error rate is something like 60% to 70%. Still, it seems like the groundhog is more often wrong than right.