I recently switched this blog from a full-text feed to a feed which shows just the beginning of each post. The reason I did this was in order to encourage more people to come to the actual site, which I am hoping will encourage more comments and discussion. I'm open to going back to the old format, though. If you have any thoughts on this matter please let me know.
edited, 6:02 pm: I've gone back to the full-text feed.
31 July 2007
30 July 2007
seven trees in one! and how laziness can be bad.
Sum divergent series, II, from The Everything Seminar. We learn
(1 + 1 + 2 + 5 + 14 + 42 + ...) = (1-√(-3))/2
in some sense (this follows from the usual generating function for the Catalan numbers, by letting z approach 1). This is a bit surprising, because we add a bunch of positive numbers and get something which isn't even real, but weird things happen with infinity. It turns out that this is a sixth root of unity, and therefore in some sense
(1 + 1 + 2 + 5 + 14 + 42 + ...)7 = (1 + 1 + 2 + 5 + 14 + 42 + ...)
which suggests that there ought to be a simple bijection between 7-tuples of trees (counted on the left) and trees (counted on the right). What that means is that we should be able take seven trees and in some way make one big tree out of them, and that furthermore this scheme is reversible -- that is, given the one big tree, we can determine which seven small trees it came from. And, in fact, there is! You can read about it here. The actual theorem is that "There is a very explicit bijection between the set of all seven-tuples of trees and the set of all trees", where "very explicit" has a certain technical sense. The bijection is fairly simple (see page 3) and takes about a quarter-page to specify; proving that is in, in fact, a bijection takes a page; the rest of the paper is taken up with describing why anyone would think such a crazy thing would exist in the first place.
A divergent series that comes up a lot in combinatorics is
P(z) = 1 + z + 2z2 + 6z3 + 24z4 + 120z5 + ...
where the coefficients are the factorials; I don't know of any way to assign a meaningful number to this sum, although one might exist. It is the generating function of the permutations; that is, the coefficient of zn is the number of permutations of [1, 2, ..., n], or the number of ways in which we can write the numbers from 1 to n in some order. It turns out that although this series is divergent, we can calculate with this in a useful way. For example, if we want to know the number of so-called indecomposable permutations, we can write
I(z) = 1 - 1/P(z)
where the division is in the sense of a formal power series, and then I(z) turns out to be the generating function of the indecomposable permutations. (The details can be found on p. 82 of Analytic Combinatorics by Flajolet and Sedgewick; the book is still in preparation, so the page number might change.) It would be possible to do this without using divergent generating functions -- but a lot harder. This is true of a lot of results with generating functions -- to transform a lot of basic combinatorics into routine computations. The advantage of this is clear -- by making problems that used to be hard into routine exercises, that frees up people to do more work that builds on those exercises. It's often been said that mathematicians are naturally lazy, which is a statement that might seem quite strange to the layperson. But what this means is that we try to come up with ways to automate a computation so that we don't actually have to do it ourselves. The unfortunate byproduct of this is that it often seems that doing the actual computation is useful for getting a feeling for the problem. If I were actually doing some work that involved indecomposable permutations, I would almost certainly write them out for, say, n=3 and n=4; that would give me more of an intuitive sense for the problem, even if it would be a very inefficient way to work if I just wanted to know how many of them there are.
(1 + 1 + 2 + 5 + 14 + 42 + ...) = (1-√(-3))/2
in some sense (this follows from the usual generating function for the Catalan numbers, by letting z approach 1). This is a bit surprising, because we add a bunch of positive numbers and get something which isn't even real, but weird things happen with infinity. It turns out that this is a sixth root of unity, and therefore in some sense
(1 + 1 + 2 + 5 + 14 + 42 + ...)7 = (1 + 1 + 2 + 5 + 14 + 42 + ...)
which suggests that there ought to be a simple bijection between 7-tuples of trees (counted on the left) and trees (counted on the right). What that means is that we should be able take seven trees and in some way make one big tree out of them, and that furthermore this scheme is reversible -- that is, given the one big tree, we can determine which seven small trees it came from. And, in fact, there is! You can read about it here. The actual theorem is that "There is a very explicit bijection between the set of all seven-tuples of trees and the set of all trees", where "very explicit" has a certain technical sense. The bijection is fairly simple (see page 3) and takes about a quarter-page to specify; proving that is in, in fact, a bijection takes a page; the rest of the paper is taken up with describing why anyone would think such a crazy thing would exist in the first place.
A divergent series that comes up a lot in combinatorics is
P(z) = 1 + z + 2z2 + 6z3 + 24z4 + 120z5 + ...
where the coefficients are the factorials; I don't know of any way to assign a meaningful number to this sum, although one might exist. It is the generating function of the permutations; that is, the coefficient of zn is the number of permutations of [1, 2, ..., n], or the number of ways in which we can write the numbers from 1 to n in some order. It turns out that although this series is divergent, we can calculate with this in a useful way. For example, if we want to know the number of so-called indecomposable permutations, we can write
I(z) = 1 - 1/P(z)
where the division is in the sense of a formal power series, and then I(z) turns out to be the generating function of the indecomposable permutations. (The details can be found on p. 82 of Analytic Combinatorics by Flajolet and Sedgewick; the book is still in preparation, so the page number might change.) It would be possible to do this without using divergent generating functions -- but a lot harder. This is true of a lot of results with generating functions -- to transform a lot of basic combinatorics into routine computations. The advantage of this is clear -- by making problems that used to be hard into routine exercises, that frees up people to do more work that builds on those exercises. It's often been said that mathematicians are naturally lazy, which is a statement that might seem quite strange to the layperson. But what this means is that we try to come up with ways to automate a computation so that we don't actually have to do it ourselves. The unfortunate byproduct of this is that it often seems that doing the actual computation is useful for getting a feeling for the problem. If I were actually doing some work that involved indecomposable permutations, I would almost certainly write them out for, say, n=3 and n=4; that would give me more of an intuitive sense for the problem, even if it would be a very inefficient way to work if I just wanted to know how many of them there are.
Danica McKellar's "Math Doesn't Suck", and Erdos-Bacon numbers
Danica McKellar, of Wonder Years fame, has written a book called Math Doesn't Suck: How to Survive Middle-School Math Without Losing Your Mind or Breaking a Nail. Like a lot of non-fiction books these days, it has its own web site, mathdoesntsuck.com. As some of you probably know, McKellar majored in mathematics at UCLA, and is apparently reasonably good at it; she co-wrote a paper while an undergrad, which isn't exactly common. (Thanks to Jessica Gold Haralson, who let me know about this.) The book comes out this Friday; see cnn.com or jezebel. The underlying message of the book appears to be that, well, math doesn't suck. From what I've heard from various people I've talked to, a lot of people lose interest in math sometime in middle school (or in high school); the two most common reasons seem to be either that they thought showing an interest in math would make them less popular, or that they had a bad math teacher. Since math, especially as it is taught in the schools, has such a sequential nature -- you can't learn a given piece of math without knowing a large fraction of the stuff that came before it -- a single bad math teacher, or a single year spent thinking that math sucks -- can doom a lot of people. (This is in contrast to, say, spending a year not giving a damn about English class; my instinct is that it would be a lot easier to catch up there, because what one learns in English class in year N+1 doesn't depend that strongly on what one learned in year N. I admit that I may be a bit biased here, though; it is natural for me to try to convince myself that my field is more intellectually demanding than other people's fields, because then I feel better about myself.)The book itself appears to be a mixture of mathematics tips and motivational prose; I will refrain from commenting on the book, because I haven't read it. There are fully-worked-out solutions to the problems in the book, which might help you figure out what mathematics it covers; it appears to include the highlights of a pretty standard middle-school mathematics curriculum in a U. S. school. McKellar has focused some of her philanthropic efforts on the importance of quality education, including the Figure this! campaign; her web page also includes a section where she offers math help to people who write in with questions. (Will we ever see a mathematician who offers acting tips on their web page?)
McKellar is one of the eighteen or so people with a finite Erdos-Bacon number. A person's Erdos number is defined to be zero if they are Paul Erdos; otherwise it is the minimum of the Erdos numbers of the people with whom they have cowritten scholarly papers, plus one. More informally, it is the length of the shortest chain of collaboration that connected a person to Erdos. If a person can not be connected to Erdos by such a chain, their Erdos number is said to be infinite. Bacon numbers are defined similarly, but Erdos is replaced by Kevin Bacon and "have cowritten a scholarly paper" is replaced by "have appeared in the same movie". A person's Erdos-Bacon number is the sum of their Erdos number and their Bacon number; thus to have a finite such number a person needs to have appeared in a movie and written an academic paper. The canonical source on Erdos numbers is Jerry Grossman's Erdos number project; I'm not aware of a single canonical source for Bacon numbers.
Another person with an Erdos-Bacon number who's been in the news a lot lately is, believe it or not, Hank Aaron, he of the 755 home runs -- although this one is a bit facetious; Hank Aaron and Paul Erdos both autographed the same baseball, giving him an Erdos number of 1; Hank Aaron also appeared in the baseball movie Summer Catch as himself, thus connecting him to the acting world. It turns out his Bacon number is 2, giving him an Erdos-Bacon number of 3. (Some people claim that appearances as oneself don't count towards Bacon numbers.)
If you don't count people who appeared as extras or as themselves -- but only people who are credited as appearing as someone other than themselves -- the lowest Erdos-Bacon number is either 4 or 5, and it belongs to Dave Bayer, a mathematician who played a small role in A Beautiful Mind.
Language Log dissects science journalism
From Language Log: Two simple numbers and Thou shalt not report odds ratios by Mark Liberman.
The first of these, from a week ago, suggests the following rule:
This is something that I've often worried about; in one of the examples that Liberman cites, (C) and (D) are 77% and 66%.
Also, there's a link to a New York Times article (July 19) with the headline Scientists Find Genetic Link for a Disorder (Next, Respect?). Does a disease need a genetic basis in order for people to take diagnoses of it seriously? All of someone's genes are determined before they're born; this seems to imply that things which happen during a person's life which affect their health don't matter. (Please don't get me started on people who think that homosexuality is okay if, and only if, it's genetic. And even if there is a "gay gene", it's not like everyone who has it is gay and everyone who doesn't have it isn't. If the inheritance patterns for homosexuality were that simple we'd have figured it out already.
But, you know, numbers scare people. If you put numbers in a newspaper article they'll throw up their hands and turn on some reality television.
At least in the first case I had realized that there was missing information. The second of these seems more insidious to me, because I'd never thought about it before, and I'm smarter than most people about these things. (You probably are, too, if you're reading this. If you don't believe me, get out of the house some time.) A recent study was reported in the popular press with phrases such as this (from the New York Times):
As it turns out, the referral rate for white men was 90.4%, and for women and blacks 84.7%. (While I'm on the subject: conflating "women" and "blacks" like this seems kind of silly. And by "men and whites" they apparently actually meant "white men".) The study reports an "odds ratio"; the odds of a white man being referred are 9.6 to 1, and the odds of a black person or woman being referred are 15.5 to 1. The ratio of these numbers is where the 60% comes from.
The following sentence would actually be pretty close to true:
The relevant percentages are 9.6% and 15.3%, which are close enough to zero that the results don't get distorted too badly by all this manipulation. When it's put that way, it's hard to understand, but if we take not ordering catheterization as some sort of negligence you can see how it would come about. Still, it's the sort of sentence with lots of quantifiers that only a mathematician could love.
It seems that odds ratios are often given in the medical literature due to the fact that they arise more naturally from certain statistical tests. But the media has a responsibility to translate the facts into language that the hypothetical "educated layperson" can understand. And the schools have a responsibility to create "educated laypeople" who can then read such an article and understand it, but this is not a post about education.
The first of these, from a week ago, suggests the following rule:
Today's prescription is a trivial rule of scientific rhetoric. When there's a claim that some genomic variant is associated with some phenotypic trait -- whether it's breast cancer or homosexuality or conservatism or stuttering -- we need to know four simple numbers. Specifically: (A) the number of "case subjects" in the study (people with the trait in question); (B) the number of "control subjects" in the study; (C) the proportion of the case subjects with the genomic variant in question; and (D) the proportion of the controls with the genomic variant in question.
If four numbers are too many, leave out (A) and (B), as long as they're not really small. But stick with (C) and (D) -- they're the medicine that really does the work here.
This is something that I've often worried about; in one of the examples that Liberman cites, (C) and (D) are 77% and 66%.
Also, there's a link to a New York Times article (July 19) with the headline Scientists Find Genetic Link for a Disorder (Next, Respect?). Does a disease need a genetic basis in order for people to take diagnoses of it seriously? All of someone's genes are determined before they're born; this seems to imply that things which happen during a person's life which affect their health don't matter. (Please don't get me started on people who think that homosexuality is okay if, and only if, it's genetic. And even if there is a "gay gene", it's not like everyone who has it is gay and everyone who doesn't have it isn't. If the inheritance patterns for homosexuality were that simple we'd have figured it out already.
But, you know, numbers scare people. If you put numbers in a newspaper article they'll throw up their hands and turn on some reality television.
At least in the first case I had realized that there was missing information. The second of these seems more insidious to me, because I'd never thought about it before, and I'm smarter than most people about these things. (You probably are, too, if you're reading this. If you don't believe me, get out of the house some time.) A recent study was reported in the popular press with phrases such as this (from the New York Times):
Doctors are only 60% as likely to order cardiac catheterization for women and blacks as for men and whites.
As it turns out, the referral rate for white men was 90.4%, and for women and blacks 84.7%. (While I'm on the subject: conflating "women" and "blacks" like this seems kind of silly. And by "men and whites" they apparently actually meant "white men".) The study reports an "odds ratio"; the odds of a white man being referred are 9.6 to 1, and the odds of a black person or woman being referred are 15.5 to 1. The ratio of these numbers is where the 60% comes from.
The following sentence would actually be pretty close to true:
Doctors are only 60% as likely to not order cardiac catheterization for white men as for women and blacks.
The relevant percentages are 9.6% and 15.3%, which are close enough to zero that the results don't get distorted too badly by all this manipulation. When it's put that way, it's hard to understand, but if we take not ordering catheterization as some sort of negligence you can see how it would come about. Still, it's the sort of sentence with lots of quantifiers that only a mathematician could love.
It seems that odds ratios are often given in the medical literature due to the fact that they arise more naturally from certain statistical tests. But the media has a responsibility to translate the facts into language that the hypothetical "educated layperson" can understand. And the schools have a responsibility to create "educated laypeople" who can then read such an article and understand it, but this is not a post about education.
29 July 2007
links for 29 July
From The Everything Seminar: Sum Divergent Series, I". Matt suggests that
1 + 2 + 4 + 8 + 16 + 32 + ... = -1
and explains why. He promises more in this vein. As many of you might see, this formula comes from taking the well-known sum of a geometric series,
1 + x + x2 + x3 + ... = (1-x)-1
and letting x=2. This is true in the 2-adic numbers. It is also true in certain sorts of computer architectures; read item 154 of HAKMEM to find out which ones. (Read the rest of HAKMEM while you're at it. Well, except for the hardware part at the end. Hardware is stupid.)
Something that comes to mind when I think about series like this is that the Riemann zeta function is often defined in popular books (such as the various recent books on the Riemann hypothesis) as
ζ(s) = 1-s + 2-s + 3-s + 4-s + ...
and this series only converges when the real part of s is greater than 1. These books then go on to talk about how all the zeroes of the zeta function are either negative even integers (the "trivial zeroes") or are believed to have real part 1/2 (this last part being the Riemann hypothesis). Yet as far as I remember, they never point out that the definition above doesn't properly make sense for any of the values of s just mentioned; you have to analytically continue the function from the half-plane where it's already defined. There's a unique way to do this, so it's not a big problem, but it's a little bit annoying. Still, though, when you do that correctly ζ(-2) = 0. But if you look at that sum,
ζ(-2) "=" 12 + 22 + 32 + 42 + ...
So does this mean that the sum of all the squares is zero? Similarly, the sum of all the fourth powers should be ζ(-4) = 0. So the sum of all the squares which are not fourth powers must also be zero, and so on ad nauseam.
From The n-Category cafe: David Corfield writes about Rota's distinction between "Algebra 1" and "Algebra 2", roughly speaking algebraic geometry and number theory versus the more combinatorial parts of algebra. My first thought here is that it's a useful distinction to make. My second is that the names are unfortunate, because when I hear "Algebra 1" and "Algebra 2" I think of classes one typically takes in high school where one learns to manipulate linear, quadratic, etc. equations.
From Math Notations: percentages are confusing to students>
From Statistical Modeling, Causal Inference, and Social Science: a map of the results of some U.S. election (I'm not sure which one, but if I had to guess I'd say 2004 presidential) with red and blue dots representing Republican and Democratic voters; each dot represents a certain number of people. I like this idea; if you remember seeing the usual political maps there's a lot more red than blue, which seems wrong because the country is evenly split (as we saw in the 2000 and 2004 presidential elections). This seems to be a nice solution. I find myself fascinated by the fact that in the middle part of the country the population centers seem to fall on a square grid. I suspect this is due to the influence of the road system consisting mostly of north-south and east-west roads, with population centers appearing at the intersections of these roads; my instinct is that a triangular lattice would be more "efficient" somehow, although I have some difficulty articulating why. Someone speculates this is the work of Robert Vanderbei, whose web site is full of pretty pictures of things mathematical.
1 + 2 + 4 + 8 + 16 + 32 + ... = -1
and explains why. He promises more in this vein. As many of you might see, this formula comes from taking the well-known sum of a geometric series,
1 + x + x2 + x3 + ... = (1-x)-1
and letting x=2. This is true in the 2-adic numbers. It is also true in certain sorts of computer architectures; read item 154 of HAKMEM to find out which ones. (Read the rest of HAKMEM while you're at it. Well, except for the hardware part at the end. Hardware is stupid.)
Something that comes to mind when I think about series like this is that the Riemann zeta function is often defined in popular books (such as the various recent books on the Riemann hypothesis) as
ζ(s) = 1-s + 2-s + 3-s + 4-s + ...
and this series only converges when the real part of s is greater than 1. These books then go on to talk about how all the zeroes of the zeta function are either negative even integers (the "trivial zeroes") or are believed to have real part 1/2 (this last part being the Riemann hypothesis). Yet as far as I remember, they never point out that the definition above doesn't properly make sense for any of the values of s just mentioned; you have to analytically continue the function from the half-plane where it's already defined. There's a unique way to do this, so it's not a big problem, but it's a little bit annoying. Still, though, when you do that correctly ζ(-2) = 0. But if you look at that sum,
ζ(-2) "=" 12 + 22 + 32 + 42 + ...
So does this mean that the sum of all the squares is zero? Similarly, the sum of all the fourth powers should be ζ(-4) = 0. So the sum of all the squares which are not fourth powers must also be zero, and so on ad nauseam.
From The n-Category cafe: David Corfield writes about Rota's distinction between "Algebra 1" and "Algebra 2", roughly speaking algebraic geometry and number theory versus the more combinatorial parts of algebra. My first thought here is that it's a useful distinction to make. My second is that the names are unfortunate, because when I hear "Algebra 1" and "Algebra 2" I think of classes one typically takes in high school where one learns to manipulate linear, quadratic, etc. equations.
From Math Notations: percentages are confusing to students>
From Statistical Modeling, Causal Inference, and Social Science: a map of the results of some U.S. election (I'm not sure which one, but if I had to guess I'd say 2004 presidential) with red and blue dots representing Republican and Democratic voters; each dot represents a certain number of people. I like this idea; if you remember seeing the usual political maps there's a lot more red than blue, which seems wrong because the country is evenly split (as we saw in the 2000 and 2004 presidential elections). This seems to be a nice solution. I find myself fascinated by the fact that in the middle part of the country the population centers seem to fall on a square grid. I suspect this is due to the influence of the road system consisting mostly of north-south and east-west roads, with population centers appearing at the intersections of these roads; my instinct is that a triangular lattice would be more "efficient" somehow, although I have some difficulty articulating why. Someone speculates this is the work of Robert Vanderbei, whose web site is full of pretty pictures of things mathematical.
How much is it worth to win at Jeopardy!?
Most weekday evenings at 7pm I watch the television show Jeopardy! One day I'll be on the show, if I actually get around to auditioning and then I get lucky enough to get picked. For now, I just scream "how could you not know that!" from the comfort of my living room. I bet all the contestants do that.
For those of you who aren't familiar with the game, it is a game where three players competing against each other answer trivia questions (although Jeopardy! has this silly trope where your answer has to be "in the form of a question", that's entirely irrelevant here). If they get the questions right, they gain money; if they get them wrong, they lose money. Those players who have a positive amount of money after the first sixty questions (which occur in two rounds of thirty; each round has six categories of five questions, worth varying amounts) get to participate in "Final Jeopardy!". Most players end up with a positive amount of money, since they know themselves well enough to only attempt to answer questions which they think they know the correct answer to. The players are told the category of the question; they then can wager any or all of their money that they'll get it right. Then the question is stated and they have thirty seconds to write down the answer.
Fans of the show have put together a Jeopardy! archive. There's a wagering calculator available online that takes into account common wagering strategies. But this does not take into account the following facts:
The first two here, I plan to address in a later post. It's the third one I want to talk about right now.
So first we must answer the question -- what's the probability of winning one's second game, given that you've won the first? Of winning one's third game, given that you've won the first two? It's obvious that a defending champion is probably better than the average player (because they've won at least one game), but how much better?
Fortunately, the Jeopardy! Archive can help us answer that. It would be enough to know what proportion of champions win one game, two games, three games, and so on. The archive compiles an impressive set of statistics for each season, but this is not one of them, so I have to do it myself. There's a natural cutoff point in the data -- for a long time Jeopardy forced its champions to leave after five games, but they don't do that any more, since the beginning of the 2003-04. (This led to Ken Jennings' historic 75-game run in 2004.) I originally planned to go back to the beginning of that season, but they only go as far back as the beginning of the Ken Jennings run, near the end of that season.
Since June 2, 2004, there have been:
155 one-game winners
61 two-game winners
21 three-game winners
6 four-game winner
9 five-game winners
1 six-game winner
1 eight-game winner
1 nineteen-game winner
1 74-game winner
Now, if there was no effect like the one I just mentioned, you'd expect there to be three times as many one-game winners as two-game winners, two-game winners as three-game winners, and so on. It actually seems like if there is such an effect, it's not that strong. I attribute the big peak at five games to psychology; it's probably hard to win a sixth game because you're going into some sort of "unknown" territory. Notice that only four players -- Ken Jennings, David Madden, Tom Kavanaugh, and Kevin Marshall -- have won their sixth game.
So I'll assume that there is no "memory effect" -- that if you win today, you have a one-in-three chance of winning tomorrow. This seems believable -- the categories can be very different from day to day -- but I've never seen this analysis before. (It wouldn't surprise me if other Jeopardy! hopefuls have done it, though, because they seem to be That Sort Of People.)
Thus, when wagering in Final Jeopardy, one should wager as if the prize is not just the money you're going to win -- but one and a half times that much, since you can expect to win half a time more. The average champion is a one-and-a-half game winner.
But there are two problems:
- how do you use that information? Does the amount of money one expects to win really affect proper wagering strategy?
- more importantly, you only get to play at Jeopardy! once. I think the rules say that; in any case, I've never heard of somebody who's played twice on the Alex Trebek version of the show. (There are a few cases of people who played on the Alex Trebek version and on some prior version.) So anything you say about "expected value" is meaningless! What's the point of an operation that talks about the average amount you expect to win if you don't get to play long enough for that average to take effect?
For those of you who aren't familiar with the game, it is a game where three players competing against each other answer trivia questions (although Jeopardy! has this silly trope where your answer has to be "in the form of a question", that's entirely irrelevant here). If they get the questions right, they gain money; if they get them wrong, they lose money. Those players who have a positive amount of money after the first sixty questions (which occur in two rounds of thirty; each round has six categories of five questions, worth varying amounts) get to participate in "Final Jeopardy!". Most players end up with a positive amount of money, since they know themselves well enough to only attempt to answer questions which they think they know the correct answer to. The players are told the category of the question; they then can wager any or all of their money that they'll get it right. Then the question is stated and they have thirty seconds to write down the answer.
Fans of the show have put together a Jeopardy! archive. There's a wagering calculator available online that takes into account common wagering strategies. But this does not take into account the following facts:
- only the player who wins the game gets to keep their money (probably an average of $20,000 or so); the second- and third-place players get $2,500 and $1,000 respectively. (Incidentally, the show's host, Alex Trebek, seems to refer to in-game totals as "points", which I suspect is tied to this;
- perhaps wagering strategy should depend on how likely one thinks one is to get the question right, and how likely one thinks one's opponents are
- the player who wins get to come back the next day; the others don't.
The first two here, I plan to address in a later post. It's the third one I want to talk about right now.
So first we must answer the question -- what's the probability of winning one's second game, given that you've won the first? Of winning one's third game, given that you've won the first two? It's obvious that a defending champion is probably better than the average player (because they've won at least one game), but how much better?
Fortunately, the Jeopardy! Archive can help us answer that. It would be enough to know what proportion of champions win one game, two games, three games, and so on. The archive compiles an impressive set of statistics for each season, but this is not one of them, so I have to do it myself. There's a natural cutoff point in the data -- for a long time Jeopardy forced its champions to leave after five games, but they don't do that any more, since the beginning of the 2003-04. (This led to Ken Jennings' historic 75-game run in 2004.) I originally planned to go back to the beginning of that season, but they only go as far back as the beginning of the Ken Jennings run, near the end of that season.
Since June 2, 2004, there have been:
155 one-game winners
61 two-game winners
21 three-game winners
6 four-game winner
9 five-game winners
1 six-game winner
1 eight-game winner
1 nineteen-game winner
1 74-game winner
Now, if there was no effect like the one I just mentioned, you'd expect there to be three times as many one-game winners as two-game winners, two-game winners as three-game winners, and so on. It actually seems like if there is such an effect, it's not that strong. I attribute the big peak at five games to psychology; it's probably hard to win a sixth game because you're going into some sort of "unknown" territory. Notice that only four players -- Ken Jennings, David Madden, Tom Kavanaugh, and Kevin Marshall -- have won their sixth game.
So I'll assume that there is no "memory effect" -- that if you win today, you have a one-in-three chance of winning tomorrow. This seems believable -- the categories can be very different from day to day -- but I've never seen this analysis before. (It wouldn't surprise me if other Jeopardy! hopefuls have done it, though, because they seem to be That Sort Of People.)
Thus, when wagering in Final Jeopardy, one should wager as if the prize is not just the money you're going to win -- but one and a half times that much, since you can expect to win half a time more. The average champion is a one-and-a-half game winner.
But there are two problems:
- how do you use that information? Does the amount of money one expects to win really affect proper wagering strategy?
- more importantly, you only get to play at Jeopardy! once. I think the rules say that; in any case, I've never heard of somebody who's played twice on the Alex Trebek version of the show. (There are a few cases of people who played on the Alex Trebek version and on some prior version.) So anything you say about "expected value" is meaningless! What's the point of an operation that talks about the average amount you expect to win if you don't get to play long enough for that average to take effect?
28 July 2007
baseball commentators say silly things
Heard just now on FOX, which is airing the Braves-Diamondbacks game:
First, the TV commentator claims that the Diamondbacks are a very streaky team this year, because they've had three separate five-game losing streaks and have won their last seven.
In fact, the Diamondbacks have won 57 games out of 105, for a winning "percentage" of 0.543; thus their probability of losing five straight games is (1-57/105)5 = 0.0200. They've played one hundred and five games so far, so there are 101 games in which they could have started a five-game losing streak; thus their expected number of five-game losing streaks is something like (0.0200)(101) = 2.01. (Yes, that's right; the figure 0.0200 is rounded.) So it's not all that surprising that they've had three such streaks.
Similarly, the expected number of seven-game winning streaks is 99(57/105)7 = 1.37; the fact that the Diamondbacks have had one such streak is not at all surprising. (If I had to guess, I'd say that 1 is actually the most likely number of such streaks, but I'm not interested enough to do the analysis.)
Of course, not every game is independent. A more sophisticated analysis would take into account which teams were playing, and so on. An even more sophisticated analysis of streaks in baseball ought to take into account the pitching rotation; the existence of a pitching rotation reduces the likelihood of streaks. Let's say your team wins one-half of its games; then the probability of winning five straight games is 0.03125. But now say you have five starting pitchers, and your team wins in 70%, 60%, 50%, 40%, and 30% of their games respectively. If each pitcher pitches every fifth game, then the probability of winning five consecutive games is now (0.7)(0.6)(0.5)(0.4)(0.3) = 0.0252.
See The Hot Hand in Sports for more of this sort of analysis.
Second, the Braves are, according to the television guy, "exactly one percentage point" behind the Phillies. The Braves are 54-50 going into today's play; the Phillies are 53-49. Baseball winning "percentages" are conventionally reported to three decimal places; the Braves are at .519, the Phillies at .520. For those of you who don't know, it's conventional to say that one team is ahead of the other by "percentage points" in a situation such as this where both teams have the same difference between their number of wins and number of losses; in this case both teams have won four more games than they've lost. But what bothers me is the "exactly one" here; of course those figures are rounded. As it turns out, the Braves' winning percentage is 0.519230...; the Phillies; is 0.519608...; the difference is 0.000377..., or not even half a point. If baseball truncated winning percentages, instead of rounding them, the two teams would be "tied".
The first of these things -- the streakiness comment -- is the one that bothers me more, though. The "percentage points" comment is just a matter of a convention that disagrees with the one the rest of the world makes. (Why doesn't baseball report winning percentages to just two decimal places? Because that wouldn't be enough accuracy; baseball teams play 162 games a season.) But the streakiness comment is the sort of thing that shows that people don't understand the nature of randomness; people read something into "streaks" that is really just good luck.
First, the TV commentator claims that the Diamondbacks are a very streaky team this year, because they've had three separate five-game losing streaks and have won their last seven.
In fact, the Diamondbacks have won 57 games out of 105, for a winning "percentage" of 0.543; thus their probability of losing five straight games is (1-57/105)5 = 0.0200. They've played one hundred and five games so far, so there are 101 games in which they could have started a five-game losing streak; thus their expected number of five-game losing streaks is something like (0.0200)(101) = 2.01. (Yes, that's right; the figure 0.0200 is rounded.) So it's not all that surprising that they've had three such streaks.
Similarly, the expected number of seven-game winning streaks is 99(57/105)7 = 1.37; the fact that the Diamondbacks have had one such streak is not at all surprising. (If I had to guess, I'd say that 1 is actually the most likely number of such streaks, but I'm not interested enough to do the analysis.)
Of course, not every game is independent. A more sophisticated analysis would take into account which teams were playing, and so on. An even more sophisticated analysis of streaks in baseball ought to take into account the pitching rotation; the existence of a pitching rotation reduces the likelihood of streaks. Let's say your team wins one-half of its games; then the probability of winning five straight games is 0.03125. But now say you have five starting pitchers, and your team wins in 70%, 60%, 50%, 40%, and 30% of their games respectively. If each pitcher pitches every fifth game, then the probability of winning five consecutive games is now (0.7)(0.6)(0.5)(0.4)(0.3) = 0.0252.
See The Hot Hand in Sports for more of this sort of analysis.
Second, the Braves are, according to the television guy, "exactly one percentage point" behind the Phillies. The Braves are 54-50 going into today's play; the Phillies are 53-49. Baseball winning "percentages" are conventionally reported to three decimal places; the Braves are at .519, the Phillies at .520. For those of you who don't know, it's conventional to say that one team is ahead of the other by "percentage points" in a situation such as this where both teams have the same difference between their number of wins and number of losses; in this case both teams have won four more games than they've lost. But what bothers me is the "exactly one" here; of course those figures are rounded. As it turns out, the Braves' winning percentage is 0.519230...; the Phillies; is 0.519608...; the difference is 0.000377..., or not even half a point. If baseball truncated winning percentages, instead of rounding them, the two teams would be "tied".
The first of these things -- the streakiness comment -- is the one that bothers me more, though. The "percentage points" comment is just a matter of a convention that disagrees with the one the rest of the world makes. (Why doesn't baseball report winning percentages to just two decimal places? Because that wouldn't be enough accuracy; baseball teams play 162 games a season.) But the streakiness comment is the sort of thing that shows that people don't understand the nature of randomness; people read something into "streaks" that is really just good luck.
probabilities in "Set for Life"
On Friday, July 20, on ABC, a program called Set For Life premiered; another episode aired last night.
It's July. It's a prime-time game show. As you may have guessed, this is the stupidest game show ever. By "stupidest" I don't mean "the show is a bad idea" -- although I think it is, and it probably won't last long, nobody ever went broke by betting on the stupidity of the American public. But what I mean is that it involves absolutely no skill.
It reminds me of Deal or No Deal, in that no skill is involved except the still of knowing when to stop. In fact, the New York Times compared this show to Deal or No Deal back in October, writing: "On “Deal or No Deal,” Mr. Mandel does not even pretend that skill is involved. He says upfront that his game is about “giving away a ton of money,” not winning a ton of money."
Well, this is in the same vein. Here's how it works. First, there's a round that I didn't see, in which somehow it is determined how much the player will win per month; I don't know how they do this. (The Wikipedia article implies it doesn't air; people on various game show forums, like this one, seem to think that this part is probably more interesting than what actually airs. I agree, because what actually airs is less interesting than staring out my window. To be fair, staring out my window is pretty interesting.) Then, in the second round, it's determined how long the player will receive this monthly amount for. There are fifteen lights embedded in the stage; eleven are white, four are red. The player picks a light, and if it's white they go one step "up the ladder"; if it's red they go one step "down the ladder". After picking a white light, the player can elect to get out of the game and keep the money; after a red light, they are not allowed to make this choice. If they pick all four red lights, they go home with no money. The ladder contains the following time amounts:
zero, 1 month, 6 months, 1 year, 2 years, 3 years, 5 years, 10 years, 15 years, 20 years, 25 years, "set for life" (40 years).
(It's not clear to me what happens if you pick a red light on the first step. This turns out not to matter in my analysis.)
They have someone in the audience (in the first episode it was the contestant's nephew) giving "advice" on which lights to pick, and there's the pretense that some level of "skill" is involved in picking which lights are going to be white and which are going to be red, but there's actually no pattern at all. So it's really just a game of chicken. There's something of a morality play embedded in this show -- sometimes greed is good, but sometimes it can backfire and you get screwed over.
(The show's gimmick is that there's someone in an "isolation pod" and they can decide to end the game as well, but the person outside doesn't know when that's happened; the person in the isolation pod can choose to stop the game; but I'll ignore this.)
At least with Who Wants To Be A Millionaire?, which started this whole modern game-show craze, some knowledge was required. What I like about the new shows is the amount of drama that's invested in them -- each "decision" to keep going comes with dramatic music, as if it Really Matters. As if the players had any sort of control over the game. But for some reason the format of uncovering lights that have already been set works a lot better, on TV, than if there were just a random number generator simulating the light picking. For example, if there were four red lights left and seven white lights left, it would say "you win" with probability 7/11 and "you lose" with probability 4/11.
The first question that comes to mind -- what is the probability that a player wins the 40 years of monthly checks, given that they don't decide to "chicken out" at some mearlier step? To win this, the player has to pick all eleven white lights and no red lights, The probability of this is 1/C(15,4) = 1/1365.
Now, what's the right "strategy" for this game, in terms of trying to maximize your expected winnings? There are, at any point after you've picked a white light, two things you can do -- stop or keep going. If you "keep going", then we want to know what the probability of you being at various points on the ladder after Let's say, for example, that you've found six white lights and two red lights so far, so you're at the "2 years" position on the ladder, and five white lights and two red lights remain. You pick a light at random. The probability is 5/7 that it's white, and so now you end up with "3 years" winnings, 2/7 * 5/6 = 10/42 that you pick a red light and then a white light and end up back where you began, and 2/7 * 1/6 = 2/42 that you pick two red lights and lose everything. So the expected winnings (in years) are 3(5/7) + 2(10/42), which is greater than 2, so it's a good bet.
This same sort of reasoning works for any combination of lights. If you've so far picked zero to seven white lights and zero red lights, it turns out to be the right move to keep going. If you've picked eight white lights and zero red lights -- thus being at the "15 years" level -- staying or going are both equally good. If you've picked nine white lights and zero red lights, always stay; there are only two white lights and four red lights left! Similarly:
The usual caveat applies; calculating "expected value" is kind of meaningless when you only get to play once. People tend to be risk-averse with their winnings, so I'd expect to see people stopping before this strategy dictates it. For example, if two red lights and nine white lights have already been picked (leaving two of each), I don't see people taking the one-in-six risk of picking both of those red lights and losing it all.
It's July. It's a prime-time game show. As you may have guessed, this is the stupidest game show ever. By "stupidest" I don't mean "the show is a bad idea" -- although I think it is, and it probably won't last long, nobody ever went broke by betting on the stupidity of the American public. But what I mean is that it involves absolutely no skill.
It reminds me of Deal or No Deal, in that no skill is involved except the still of knowing when to stop. In fact, the New York Times compared this show to Deal or No Deal back in October, writing: "On “Deal or No Deal,” Mr. Mandel does not even pretend that skill is involved. He says upfront that his game is about “giving away a ton of money,” not winning a ton of money."
Well, this is in the same vein. Here's how it works. First, there's a round that I didn't see, in which somehow it is determined how much the player will win per month; I don't know how they do this. (The Wikipedia article implies it doesn't air; people on various game show forums, like this one, seem to think that this part is probably more interesting than what actually airs. I agree, because what actually airs is less interesting than staring out my window. To be fair, staring out my window is pretty interesting.) Then, in the second round, it's determined how long the player will receive this monthly amount for. There are fifteen lights embedded in the stage; eleven are white, four are red. The player picks a light, and if it's white they go one step "up the ladder"; if it's red they go one step "down the ladder". After picking a white light, the player can elect to get out of the game and keep the money; after a red light, they are not allowed to make this choice. If they pick all four red lights, they go home with no money. The ladder contains the following time amounts:
zero, 1 month, 6 months, 1 year, 2 years, 3 years, 5 years, 10 years, 15 years, 20 years, 25 years, "set for life" (40 years).
(It's not clear to me what happens if you pick a red light on the first step. This turns out not to matter in my analysis.)
They have someone in the audience (in the first episode it was the contestant's nephew) giving "advice" on which lights to pick, and there's the pretense that some level of "skill" is involved in picking which lights are going to be white and which are going to be red, but there's actually no pattern at all. So it's really just a game of chicken. There's something of a morality play embedded in this show -- sometimes greed is good, but sometimes it can backfire and you get screwed over.
(The show's gimmick is that there's someone in an "isolation pod" and they can decide to end the game as well, but the person outside doesn't know when that's happened; the person in the isolation pod can choose to stop the game; but I'll ignore this.)
At least with Who Wants To Be A Millionaire?, which started this whole modern game-show craze, some knowledge was required. What I like about the new shows is the amount of drama that's invested in them -- each "decision" to keep going comes with dramatic music, as if it Really Matters. As if the players had any sort of control over the game. But for some reason the format of uncovering lights that have already been set works a lot better, on TV, than if there were just a random number generator simulating the light picking. For example, if there were four red lights left and seven white lights left, it would say "you win" with probability 7/11 and "you lose" with probability 4/11.
The first question that comes to mind -- what is the probability that a player wins the 40 years of monthly checks, given that they don't decide to "chicken out" at some mearlier step? To win this, the player has to pick all eleven white lights and no red lights, The probability of this is 1/C(15,4) = 1/1365.
Now, what's the right "strategy" for this game, in terms of trying to maximize your expected winnings? There are, at any point after you've picked a white light, two things you can do -- stop or keep going. If you "keep going", then we want to know what the probability of you being at various points on the ladder after Let's say, for example, that you've found six white lights and two red lights so far, so you're at the "2 years" position on the ladder, and five white lights and two red lights remain. You pick a light at random. The probability is 5/7 that it's white, and so now you end up with "3 years" winnings, 2/7 * 5/6 = 10/42 that you pick a red light and then a white light and end up back where you began, and 2/7 * 1/6 = 2/42 that you pick two red lights and lose everything. So the expected winnings (in years) are 3(5/7) + 2(10/42), which is greater than 2, so it's a good bet.
This same sort of reasoning works for any combination of lights. If you've so far picked zero to seven white lights and zero red lights, it turns out to be the right move to keep going. If you've picked eight white lights and zero red lights -- thus being at the "15 years" level -- staying or going are both equally good. If you've picked nine white lights and zero red lights, always stay; there are only two white lights and four red lights left! Similarly:
- if one red light has already been picked, and eight or less white lights have been picked, it makes sense to keep going; if nine or more white lights have been picked, stop.
- if two red lights have been picked, go if nine white lights or less have been picked; stop if ten white lights have been picked.
- if three red lights have been picked, keep going if seven or less white lights have been picked; if eight or nine have been picked, staying and going are equally valuable; if ten white lights have been picked, stop.
The usual caveat applies; calculating "expected value" is kind of meaningless when you only get to play once. People tend to be risk-averse with their winnings, so I'd expect to see people stopping before this strategy dictates it. For example, if two red lights and nine white lights have already been picked (leaving two of each), I don't see people taking the one-in-six risk of picking both of those red lights and losing it all.
27 July 2007
checks for nothing, and why English is useful
Karl Fogel attempts to pay a bill for $0, because of course you have to pay bills for zero, because otherwise the companies that issue them keep sending them.
In this case, the bill was for the purchase of a book from the Mathematical Association of America, so he wrote a check for eiπ+1 dollars. They didn't deposit it, because "check needs to be wrote out in U. S. dollars", as they put it.
I can see a more legitimate reason for rejecting the check. Usually, when writing a check, one puts, say, "3.14" in the little box on the right, and "Three and 14/100" on the line. The reason for writing out the value of the check in both figured and words is for redundancy. (Although then why don't we write "three dollars and fourteen cents" on the line? I suppose redundancy doesn't matter quite as much when we're talking about sub-dollar amounts.)
So you might say he should have written "e to the i π plus one dollars" on the line. But even that seems a bit suspect, because e, i, and π are themselves bits of mathematical notation. It seems that he really should have written something like
"The base of the exponential function, raised to the product of the imaginary unit and the ratio of a circle's circumference to its diameter, plus one"
for the number of dollars he wanted. Of course, this is the sort of thing that makes it obvious why having a compact mathematical notation is a good idea. I am not enough of a mathematical historian to have looked at the way things used to be written, but from what I understand this is the sort of thing they would have written five centuries ago, and I can't imagine working like that.
Unfortunately, the fact that we have such a good mathematical notation creates another problem -- people think that they can just put a bunch of symbols on a page and not explain what they mean by them, and that's "mathematics". Terry Tao, at his blog, has lots of writing advice; of particular interest in this discussion is his advice to take advantage of the English language. Here he gives a couple dozen ways to say that two statements are true, which are logically equivalent but have a wide variety of connotations. To take two examples of his examples at random, "P(x) is true. Unfortunately, Q(y) is also true." and "P is satisfied by x. Similarly, Q is satisfied by y." might be logically equivalent but are philosophically (psychologically, emotionally, morally -- what's the right word here?) quite distinct. I think that mathematicians as a whole are not sensitive enough to the connotations of their words; this is useful when doing formal mathematics but not so useful when trying to express the results of it. Perhaps we kneel too much at the altar of Bourbaki.
In this case, the bill was for the purchase of a book from the Mathematical Association of America, so he wrote a check for eiπ+1 dollars. They didn't deposit it, because "check needs to be wrote out in U. S. dollars", as they put it.
I can see a more legitimate reason for rejecting the check. Usually, when writing a check, one puts, say, "3.14" in the little box on the right, and "Three and 14/100" on the line. The reason for writing out the value of the check in both figured and words is for redundancy. (Although then why don't we write "three dollars and fourteen cents" on the line? I suppose redundancy doesn't matter quite as much when we're talking about sub-dollar amounts.)
So you might say he should have written "e to the i π plus one dollars" on the line. But even that seems a bit suspect, because e, i, and π are themselves bits of mathematical notation. It seems that he really should have written something like
"The base of the exponential function, raised to the product of the imaginary unit and the ratio of a circle's circumference to its diameter, plus one"
for the number of dollars he wanted. Of course, this is the sort of thing that makes it obvious why having a compact mathematical notation is a good idea. I am not enough of a mathematical historian to have looked at the way things used to be written, but from what I understand this is the sort of thing they would have written five centuries ago, and I can't imagine working like that.
Unfortunately, the fact that we have such a good mathematical notation creates another problem -- people think that they can just put a bunch of symbols on a page and not explain what they mean by them, and that's "mathematics". Terry Tao, at his blog, has lots of writing advice; of particular interest in this discussion is his advice to take advantage of the English language. Here he gives a couple dozen ways to say that two statements are true, which are logically equivalent but have a wide variety of connotations. To take two examples of his examples at random, "P(x) is true. Unfortunately, Q(y) is also true." and "P is satisfied by x. Similarly, Q is satisfied by y." might be logically equivalent but are philosophically (psychologically, emotionally, morally -- what's the right word here?) quite distinct. I think that mathematicians as a whole are not sensitive enough to the connotations of their words; this is useful when doing formal mathematics but not so useful when trying to express the results of it. Perhaps we kneel too much at the altar of Bourbaki.
everything happens somewhere
With Tools on Web, Amateurs Reshape Mapmaking -- today's New York Times. (I think you have to register, but it's free.)
The headline basically says what the article's about, and points me to some things I didn't know about -- for example, Flickr, the photo-sharing service, now allows people to tag their photos with information about their location.
It'll be interesting to see which locations are overrepresented and which are underrepresented in the ones where people take photos. I've spent a fair bit of time Googling various intersections in Philadelphia and seeing which ones get a lot of hits; obviously intersections which are landmarks of one sort or another get a lot of hits, but even intersections of two quiet residential streets will have vastly differing numbers of hits, depending on -- it seems -- how wired the neighborhood in question is. It appears to not just be a question of socioeconomic status, but also of the age of the people living in the neighborhood; neighborhoods with lots of young adults have a higher profile on the web, which isn't surprising. Also, Philadelphia seems to be a good city in which to do this, because the streets form a grid and so most intersections are at least nominally equivalent.
My favorite among these mapmaking services is the simplest one I know of -- the gmaps pedometer, which allows you to overlay a walking route on a map and find out how long it is. Since I walk everywhere this is somewhat valuable. However, I wish that it were possible to get directions on some mapping service that didn't respect one-way streets, avoided highways, and so on. A friend of mine who just got a new apartment wrote recently:
But in general, everything happens somewhere, and giving people the ability to harness that fact can only be a Good Idea.
I think there will be another revolution, though. Google, Microsoft, etc. are working on these technologies to allow people to connect online data with real-life locations. But so much of what happens right now isn't really wedded to any location, and that will only continue in the future. This blog, for example, physically exists on a server somewhere... I don't know where. Oddly enough, I am reasonably sure it is not at 365 Main, a data center in San Francisco, because they had a power failure Tuesday afternoon and Blogger was still up. A lot of heavily trafficked sites were down, though, including Craigslist, Technorati, and Livejournal. It was a bit strange to see that sites which had nothing to do with each other in the virtual world nevertheless were tied together by their location in the physical world. But I don't care where the server is. (It is probably more accurate to say that insofar as my blog has a physical location, it is my kitchen table, because that's where I do most of my writing. But it is probably even more accurate to say that my blog's physical location is wherever my brain happens to be at any given moment.) What I care about is how it relates to the other information out there on the web, which I can get from, for example, seeing the "reactions" page at Technorati, looking at the service which tracks the hits this web page gets, noticing the blogs that people who link to my blog also link to, and so on. That tells me what's close to me in this virtual space.
And I wonder if this virtual space will turn into a physical space, if we will find a way to make a picture of it that makes sense. (A primitive example of what I'm thinking of is given by A Subway Map of Web Trends 2.0, from Strange Maps. Unfortunately this map is hampered by the fact that the makers didn't do their own graphic design, but rather based it on a map of the Tokyo subway system, and there's no reason that the Web should be isomorphic to the Tokyo subway. In fact, that would mean that the Web looks a lot like the actual city of Tokyo.) So far we have lists of what's close to each other. But a list of distances is not a map. Our minds are very good at making sense of spatial information, probably for evolutionary reasons; this is probably why one of the first things we tell our students to do, when it's at all relevant, is to draw a picture. But right now we only have the means to draw primitive pictures. That will change.
The headline basically says what the article's about, and points me to some things I didn't know about -- for example, Flickr, the photo-sharing service, now allows people to tag their photos with information about their location.
It'll be interesting to see which locations are overrepresented and which are underrepresented in the ones where people take photos. I've spent a fair bit of time Googling various intersections in Philadelphia and seeing which ones get a lot of hits; obviously intersections which are landmarks of one sort or another get a lot of hits, but even intersections of two quiet residential streets will have vastly differing numbers of hits, depending on -- it seems -- how wired the neighborhood in question is. It appears to not just be a question of socioeconomic status, but also of the age of the people living in the neighborhood; neighborhoods with lots of young adults have a higher profile on the web, which isn't surprising. Also, Philadelphia seems to be a good city in which to do this, because the streets form a grid and so most intersections are at least nominally equivalent.
My favorite among these mapmaking services is the simplest one I know of -- the gmaps pedometer, which allows you to overlay a walking route on a map and find out how long it is. Since I walk everywhere this is somewhat valuable. However, I wish that it were possible to get directions on some mapping service that didn't respect one-way streets, avoided highways, and so on. A friend of mine who just got a new apartment wrote recently:
Our apartment is at [street address] It's close to the T! Here are Google Maps' directions if you are a car: [link] . If you are not a car, I recommend walking from the T [...]The Google Maps directions put her new apartment at six-tenths of a mile from the T; on foot, it looks to be more like three-tenths of a mile.
But in general, everything happens somewhere, and giving people the ability to harness that fact can only be a Good Idea.
I think there will be another revolution, though. Google, Microsoft, etc. are working on these technologies to allow people to connect online data with real-life locations. But so much of what happens right now isn't really wedded to any location, and that will only continue in the future. This blog, for example, physically exists on a server somewhere... I don't know where. Oddly enough, I am reasonably sure it is not at 365 Main, a data center in San Francisco, because they had a power failure Tuesday afternoon and Blogger was still up. A lot of heavily trafficked sites were down, though, including Craigslist, Technorati, and Livejournal. It was a bit strange to see that sites which had nothing to do with each other in the virtual world nevertheless were tied together by their location in the physical world. But I don't care where the server is. (It is probably more accurate to say that insofar as my blog has a physical location, it is my kitchen table, because that's where I do most of my writing. But it is probably even more accurate to say that my blog's physical location is wherever my brain happens to be at any given moment.) What I care about is how it relates to the other information out there on the web, which I can get from, for example, seeing the "reactions" page at Technorati, looking at the service which tracks the hits this web page gets, noticing the blogs that people who link to my blog also link to, and so on. That tells me what's close to me in this virtual space.
And I wonder if this virtual space will turn into a physical space, if we will find a way to make a picture of it that makes sense. (A primitive example of what I'm thinking of is given by A Subway Map of Web Trends 2.0, from Strange Maps. Unfortunately this map is hampered by the fact that the makers didn't do their own graphic design, but rather based it on a map of the Tokyo subway system, and there's no reason that the Web should be isomorphic to the Tokyo subway. In fact, that would mean that the Web looks a lot like the actual city of Tokyo.) So far we have lists of what's close to each other. But a list of distances is not a map. Our minds are very good at making sense of spatial information, probably for evolutionary reasons; this is probably why one of the first things we tell our students to do, when it's at all relevant, is to draw a picture. But right now we only have the means to draw primitive pictures. That will change.
26 July 2007
Greater-Than Sudoku
(I apologize if this post is giving you trouble when you read it; for some reason Blogger doesn't seem to like the large numbers of inequality signs in this post.)
A sort of puzzle that I find incredibly frustrating is the "Greater-Than Sudoku".
This is like Sudoku, except that none of the numbers are given. But greater-than or less-than sign indicate which of any two adjacent squares contains the greater number. (The adjacencies are only within the 3-by-3 squares that make up the larger 9-by-9 square, so there's no possiblity that these could be equal.)
An example can be found about halfway down at Ed Pegg Jr's column, "Sudoku Variations", which appears to be part of his approximately monthly column Math Games at MAA Online. Not surprisingly, this gives a wide variety of Sudoku variations. Personally I prefer the variant called "Sums Sudoku", in which the 9-by-9 grid is broken up into regions containing a few cells and the sum of the numbers in each of these is given. I went through a whole book of those last summer. They were fun.
There's a greater-than sudoku in the July 26 issue of the Philadelphia City Paper, by Matt Jones, the author of "Psycho Sudoku", who believe it or not is not the guy behind Jonesin' crosswords, which that paper also includes. (Jonesin' crosswords are by Matt Gaffney.) You can find a whole bunch of Jones' variant Sudokus at the Boston Phoenix. These include Greater-Than Sudoku, Sum Sudoku, Stepping-Stone Sudoku, and Kaidoku, which really has nothing to do with Sudoku (it's a word puzzle) but has a similar-sounding name. The reason I like the first three of these is because they require more mathematical thinking than the standard Sudoku, which is exactly the reason that most people probably like them less. I have often found things explaining Sudoku to people who haven't seen it yet that say "it doesn't involve any math, just logic" -- and that's true, at least with the common understanding of those words.
The problem with not being given any numbers, as in the Greater-Than Sudoku, is that there's nowhere, really, to start. The way I've usually started these puzzles is as follows: the number 9 must be in a square that contains a number greater than that in each of the adjacent squares. That allows one to rule out most squares from containing 9; if you're lucky, it rules out enough squares that, combining this with the rule that there is exactly one 9 in each row, column, and 3-by-3 square, you know where all the 9's are. Replacing "greater than" with "less than", you can determine where the 1's go. Next, the 8's can be placed -- they must be in squares that are greater than everything adajcent to them, except they could be less than an adjacent 9. This seems to lead to more possible places where 8's can go, in general, than 9's, so the "ruling out" is harder. By symmetry, one can do the same thing with the 2's. My theory is that if I could follow this far enough, I would have enough numbers that I could solve basically like a regular Sudoku puzzle; unfortunately this doesn't seem to work. (And even when it does seem to be working, it requires scratch paper -- it gets too messy to fit all the notes I want to make in the squares -- which just feels wrong for a newspaper puzzle.)
Another method that has occurred to me is to try to determine what range the number in each square falls in. For example, consider the bottom-left 3-by-3 square in the Greater-Than Sudoku in Ed Pegg's article. It has
where v and ^ are meant to be downward-pointing and upward-pointing analogs of < and >, and A, B, ..., I are 1, 2, ..., 9 in some order. We have, of course, the twelve inequalities that are given explicitly: B<A, B<C, D<A, B<E, F<C, D<E, E<F, G<D, H<E, I<F, G<H, I<H. These specify a partial order of A, ..., I (for those who don't know what this means: it just means that we know some of them are larger than others, and that if x is larger than y and y is larger than z, then x is larger than z); what we want to do is extend this to a total order. If we knew, for example, that G<D<E<F<C<B<H<I<A (incidentally, this isn't true), then we'd have G = 1, D = 2, E = 3, ..., A = 9.
Since the relation "less than" is transitive, we end up with various "chains" of inequalities, the longest of which is G<D<E<F<C. We also know that G<A (since G<D and D<A). Finally, G is less than H, and we can't compare G with B or I. So there are six numbers greater than G; thus G is at most 3. I suspect that doing this sort of reasoning for each square would help.
However, the twelve comparison signs in any 3-by-3 square are not themselves enough to specify how the numbers 1, ..., 9 are arranged. There are only 212 = 4096 ways we can set them up, and 9! = 362880 ways to arrange the integers 1, ..., 9 in those boxes; on average, if we arrange the < and > signs randomly, there will be 9!/212 = 88.59375 ways to fill in the numbers 1 through 9 in that square which are consistent with that choice of inequality signs. And it is clearly possible for there to be many less. For example, we can never have a square with
(the rest being irrelevant), since we have A<D<E<B<A and thus A<A. Thus, to offset this, in some cases there will be many more; we need the Sudoku constraints to help fit things together. The total number of Sudoku puzzles is widely reported to be 6,670,903,752,021,072,936,960 (see Russell and Jarvis, which also gives a nice heuristic that gets very close); this is about 272, so it's at least plausible that the 108 comparison signs in a 9-by-9 Greater-Than Sudoku give enough information to specify a unique solution.
A sort of puzzle that I find incredibly frustrating is the "Greater-Than Sudoku".
This is like Sudoku, except that none of the numbers are given. But greater-than or less-than sign indicate which of any two adjacent squares contains the greater number. (The adjacencies are only within the 3-by-3 squares that make up the larger 9-by-9 square, so there's no possiblity that these could be equal.)
An example can be found about halfway down at Ed Pegg Jr's column, "Sudoku Variations", which appears to be part of his approximately monthly column Math Games at MAA Online. Not surprisingly, this gives a wide variety of Sudoku variations. Personally I prefer the variant called "Sums Sudoku", in which the 9-by-9 grid is broken up into regions containing a few cells and the sum of the numbers in each of these is given. I went through a whole book of those last summer. They were fun.
There's a greater-than sudoku in the July 26 issue of the Philadelphia City Paper, by Matt Jones, the author of "Psycho Sudoku", who believe it or not is not the guy behind Jonesin' crosswords, which that paper also includes. (Jonesin' crosswords are by Matt Gaffney.) You can find a whole bunch of Jones' variant Sudokus at the Boston Phoenix. These include Greater-Than Sudoku, Sum Sudoku, Stepping-Stone Sudoku, and Kaidoku, which really has nothing to do with Sudoku (it's a word puzzle) but has a similar-sounding name. The reason I like the first three of these is because they require more mathematical thinking than the standard Sudoku, which is exactly the reason that most people probably like them less. I have often found things explaining Sudoku to people who haven't seen it yet that say "it doesn't involve any math, just logic" -- and that's true, at least with the common understanding of those words.
The problem with not being given any numbers, as in the Greater-Than Sudoku, is that there's nowhere, really, to start. The way I've usually started these puzzles is as follows: the number 9 must be in a square that contains a number greater than that in each of the adjacent squares. That allows one to rule out most squares from containing 9; if you're lucky, it rules out enough squares that, combining this with the rule that there is exactly one 9 in each row, column, and 3-by-3 square, you know where all the 9's are. Replacing "greater than" with "less than", you can determine where the 1's go. Next, the 8's can be placed -- they must be in squares that are greater than everything adajcent to them, except they could be less than an adjacent 9. This seems to lead to more possible places where 8's can go, in general, than 9's, so the "ruling out" is harder. By symmetry, one can do the same thing with the 2's. My theory is that if I could follow this far enough, I would have enough numbers that I could solve basically like a regular Sudoku puzzle; unfortunately this doesn't seem to work. (And even when it does seem to be working, it requires scratch paper -- it gets too messy to fit all the notes I want to make in the squares -- which just feels wrong for a newspaper puzzle.)
Another method that has occurred to me is to try to determine what range the number in each square falls in. For example, consider the bottom-left 3-by-3 square in the Greater-Than Sudoku in Ed Pegg's article. It has
A > B < C
v ^ v
D < E < F
v v v
G < H > I
where v and ^ are meant to be downward-pointing and upward-pointing analogs of < and >, and A, B, ..., I are 1, 2, ..., 9 in some order. We have, of course, the twelve inequalities that are given explicitly: B<A, B<C, D<A, B<E, F<C, D<E, E<F, G<D, H<E, I<F, G<H, I<H. These specify a partial order of A, ..., I (for those who don't know what this means: it just means that we know some of them are larger than others, and that if x is larger than y and y is larger than z, then x is larger than z); what we want to do is extend this to a total order. If we knew, for example, that G<D<E<F<C<B<H<I<A (incidentally, this isn't true), then we'd have G = 1, D = 2, E = 3, ..., A = 9.
Since the relation "less than" is transitive, we end up with various "chains" of inequalities, the longest of which is G<D<E<F<C. We also know that G<A (since G<D and D<A). Finally, G is less than H, and we can't compare G with B or I. So there are six numbers greater than G; thus G is at most 3. I suspect that doing this sort of reasoning for each square would help.
However, the twelve comparison signs in any 3-by-3 square are not themselves enough to specify how the numbers 1, ..., 9 are arranged. There are only 212 = 4096 ways we can set them up, and 9! = 362880 ways to arrange the integers 1, ..., 9 in those boxes; on average, if we arrange the < and > signs randomly, there will be 9!/212 = 88.59375 ways to fill in the numbers 1 through 9 in that square which are consistent with that choice of inequality signs. And it is clearly possible for there to be many less. For example, we can never have a square with
A > B
^ v
D < E
(the rest being irrelevant), since we have A<D<E<B<A and thus A<A. Thus, to offset this, in some cases there will be many more; we need the Sudoku constraints to help fit things together. The total number of Sudoku puzzles is widely reported to be 6,670,903,752,021,072,936,960 (see Russell and Jarvis, which also gives a nice heuristic that gets very close); this is about 272, so it's at least plausible that the 108 comparison signs in a 9-by-9 Greater-Than Sudoku give enough information to specify a unique solution.
fundamentalists and π
Fundamentalist math, at ooblog, via Vlorbik on Math Ed. The article quotes a "fundamentalist math textbook", which apparently actually exists. It says things like:
The passage on "fundamental" seems, to me, like a real stretch; mathematicians are well-known for using words in different ways than the general population. (So are fundamentalist Christians, I hear, although I don't know this for sure.) Mathematicians actually use "fundamental" in much the same way that Normal People do; Wikipedia has a list of fundamental theorems. I would suggest one more addition to this list: the "fundamental theorem of combinatorics", which is that if there are m ways to do one thing, and then there are n ways to do some other thing, then there are mn ways to first do one of the first things and then one of the second things. I will not edit Wikipedia to reflect this, though, as this usage seems non-standard; but insofar as combinatorics has a fundamental theorem, this is it.
(Oh, by the way, if someone refers to the "fundamental theorem" without saying what it's the fundamental theorem of, what would you think they meant? I'm not sure; I'd think they meant the F. T. of either algebra, arithmetic, or calculus.)
The most famous "math in the Bible" is, of course, the verse that can be interpreted as saying that π equals 3, due to some circular object which is thirty cubits around and ten cubits across. (Link goes to Good Math, Bad Math; the Bible verse in question is 1 Kings 7:23.) Surprisingly, everyone seems to assume that this means the writer actually thought π = 3, and comes up with complicated explanations for this. For example, some people assume that 30 cubits is the inner circumference and 10 cubits is the outer diameter. I think I even once saw someone who seriously suggested that in the time of the Bible, circles were actually hexagons; the length of a regular hexagon is exactly three times the distance between opposite corners. The obvious interpretation, though, is that either:
And I'm trying to picture a world in which circles were hexagons. This seems to imply that the world is in fact a triangular lattice in the plane, which is preposterous.
Carl Friedrich Gauss first proved the fundamental theorem of algebra. There are many fundamental theorems: of arithmetic, calculus, and so on. These are so “fundamental” that many other theorems are derived from them. In the Bible, there are also fundamentals, without which Christianity would not exist—the deity of Christ, His substitutionary atonement, and the inspiration of the Bible, to name a few.Basically, this sentence says that "things, in various intellectual frameworks, are derived from other things". What else could you expect? Standing on the shoulders of giants, and all that. Something valuable for kids to know, surely, but why try so hard? I'd rather at least see a fundamentalist approach to mathematics that was based on saying "math is beautiful, and that shows us that God is great, because God invented mathematics." I wouldn't agree with this, but I could at least sympathize with it.
The passage on "fundamental" seems, to me, like a real stretch; mathematicians are well-known for using words in different ways than the general population. (So are fundamentalist Christians, I hear, although I don't know this for sure.) Mathematicians actually use "fundamental" in much the same way that Normal People do; Wikipedia has a list of fundamental theorems. I would suggest one more addition to this list: the "fundamental theorem of combinatorics", which is that if there are m ways to do one thing, and then there are n ways to do some other thing, then there are mn ways to first do one of the first things and then one of the second things. I will not edit Wikipedia to reflect this, though, as this usage seems non-standard; but insofar as combinatorics has a fundamental theorem, this is it.
(Oh, by the way, if someone refers to the "fundamental theorem" without saying what it's the fundamental theorem of, what would you think they meant? I'm not sure; I'd think they meant the F. T. of either algebra, arithmetic, or calculus.)
The most famous "math in the Bible" is, of course, the verse that can be interpreted as saying that π equals 3, due to some circular object which is thirty cubits around and ten cubits across. (Link goes to Good Math, Bad Math; the Bible verse in question is 1 Kings 7:23.) Surprisingly, everyone seems to assume that this means the writer actually thought π = 3, and comes up with complicated explanations for this. For example, some people assume that 30 cubits is the inner circumference and 10 cubits is the outer diameter. I think I even once saw someone who seriously suggested that in the time of the Bible, circles were actually hexagons; the length of a regular hexagon is exactly three times the distance between opposite corners. The obvious interpretation, though, is that either:
- The object in question wasn't perfectly round. This seems rather doubtful, though, as you just need a stake in the ground and a big long rope in order to make a circle. If they wanted to make circles, they could.
- Rounding error. What is called "30 cubits" here could be as large as 30.5 cubits; what is called "10 cubits" could actually be as small as 9.5. Thus the ratio of circumference to diameter could be as large as 30.5/9.5 = 3.21. Similarly, it could be as small as 29.5/10.5 = 2.81. So the Bible is only saying that 2.81 < π < 3.21, which is true.
And I'm trying to picture a world in which circles were hexagons. This seems to imply that the world is in fact a triangular lattice in the plane, which is preposterous.
how much should used furniture cost?
In an online forum which I frequent, someone suggested that the price of used furniture decays exponentially. So if I have a desk which I bought today for $100, and I want to sell it a year from now, it'll fetch $80; two years from now, $64, and so on, assuming that it depreciates at twenty percent per year. (The "twenty percent" is a number I just made up so that the arithmetic would be easy.)
But a flaw in this argument was quickly seen. For one thing, furniture depreciates essentially immediately the moment you take it out of the furniture-dealer's showroom. (One might make an exception for IKEA; first it depreciates, because you take it out of the store, then it appreciates because you put in some of your own labor in transforming a bewildering array of pieces of wood into something you can actually sit on, sleep in, or put your possessions in.) This was ascribed to the fact that when you go to a furniture store, you have to do a lot less work to find the particular piece that suits your needs than if you're trying to find such an item on Craigslist.
Then another model for the rate at which furniture depreciates was claimed: namely, that furniture depreciates by some fixed percentage immediately when it leaves the store, and then at a constant percentage rate per year thereafter. (The depreciation percentage upon leaving the store, at least for IKEA goods, could probably be determined by watching Craigslist; a surprisingly large amount of the IKEA furniture for sale there is said to be less than one year old. Also, a surprisingly large amount of the furniture for sale on Craigslist is IKEA; I suspect this is because the type of people who sell things at Craigslist are the same upwardly mobile, transient types that buy things at IKEA. Note that I have done absolutely no market research on this subject, unless you count "what kind of furniture do my friends have?" And I didn't even actually ask my friends, I asked myself.)
But there's a problem with this one, too. I am sitting at a nineteen-year-old kitchen table. I know it is nineteen years old because my parents bought it when they (we, since I lived with them at the time) moved into a new house in 1988. I inherited this kitchen table from them in 2005, because they moved into a house for which they wanted new furniture at the same time as I needed a table. I do not know how much they paid for it, and I do not know how much it is worth. However, I suspect that if it was worth, say, $400 when it left the showroom in 1988, and $100 right now, then it was probably worth less than $200 in 1997. (Pretend that 1997 is exactly halfway between 1988 and now.) That is, I suspect it depreciated by a larger percentage in the first half of its lifetime than in the second half. In fact, I suspect that by now this table is not depreciating at all. The exponential decay also suggests that in a reasonably short amount of time my table will be worth nothing, which also seems silly.
(Incidentally, all I can find online about furniture depreciation is information for tax purposes for furniture owned by businesses; the scheme used there assumes that furniture depreciates in a straight-line fashion over the first seven years of its life. This is so far from the obvious reality that I will not even take it into account.)
Why do I say this? Well, my table looks like it's in good shape. And let's say that you know it's twenty years old. There's no reason why it shouldn't hold up for another, say, twenty years -- the fact that it's been around this long tells you that it's built well. (I know I'm invoking the Copernican principle here, which I've criticized before in reference to space exploration, but furniture isn't space exploration.) So this should make you be willing to pay more for it; if you are buying a table, the amount you are willing to pay for it is roughly how much you would pay to have a table for a year, times how many years you expect the table to last. (If you sell the table before it stops working, this still applies, because the price you can get for it is the result of the next buyer making this same calculation.) As time goes on, the table accumulates scratches and dings -- making it worth less per year -- but its life expectancy might go up. (I know this seems somewhat paradoxical; you might think that a table has, say, a 5% chance of breaking in any given year -- a table breaking might be a Poisson process. But I'm claiming that the fact that this table is still in one piece should lead you to believe that it's a Good Table. I'm saying this mostly because it seems to lead to results about the used-furniture market that feel right.) When a piece of furniture is new this first effect is the larger one; when it is old the second effect is the larger one. At the point at which the second effect is larger, that's when the price of the thing starts going up.
People also attach a certain value to "antique" furniture, which would also raise how much they're willing to pay for it; however, I don't think my table is anywhere near old enough for that to matter.
As it turns out, I live above a used furniture store. However, I suspect that the owner of that store doesn't like me much, so I don't think I'll ask him what he thinks about this.
But a flaw in this argument was quickly seen. For one thing, furniture depreciates essentially immediately the moment you take it out of the furniture-dealer's showroom. (One might make an exception for IKEA; first it depreciates, because you take it out of the store, then it appreciates because you put in some of your own labor in transforming a bewildering array of pieces of wood into something you can actually sit on, sleep in, or put your possessions in.) This was ascribed to the fact that when you go to a furniture store, you have to do a lot less work to find the particular piece that suits your needs than if you're trying to find such an item on Craigslist.
Then another model for the rate at which furniture depreciates was claimed: namely, that furniture depreciates by some fixed percentage immediately when it leaves the store, and then at a constant percentage rate per year thereafter. (The depreciation percentage upon leaving the store, at least for IKEA goods, could probably be determined by watching Craigslist; a surprisingly large amount of the IKEA furniture for sale there is said to be less than one year old. Also, a surprisingly large amount of the furniture for sale on Craigslist is IKEA; I suspect this is because the type of people who sell things at Craigslist are the same upwardly mobile, transient types that buy things at IKEA. Note that I have done absolutely no market research on this subject, unless you count "what kind of furniture do my friends have?" And I didn't even actually ask my friends, I asked myself.)
But there's a problem with this one, too. I am sitting at a nineteen-year-old kitchen table. I know it is nineteen years old because my parents bought it when they (we, since I lived with them at the time) moved into a new house in 1988. I inherited this kitchen table from them in 2005, because they moved into a house for which they wanted new furniture at the same time as I needed a table. I do not know how much they paid for it, and I do not know how much it is worth. However, I suspect that if it was worth, say, $400 when it left the showroom in 1988, and $100 right now, then it was probably worth less than $200 in 1997. (Pretend that 1997 is exactly halfway between 1988 and now.) That is, I suspect it depreciated by a larger percentage in the first half of its lifetime than in the second half. In fact, I suspect that by now this table is not depreciating at all. The exponential decay also suggests that in a reasonably short amount of time my table will be worth nothing, which also seems silly.
(Incidentally, all I can find online about furniture depreciation is information for tax purposes for furniture owned by businesses; the scheme used there assumes that furniture depreciates in a straight-line fashion over the first seven years of its life. This is so far from the obvious reality that I will not even take it into account.)
Why do I say this? Well, my table looks like it's in good shape. And let's say that you know it's twenty years old. There's no reason why it shouldn't hold up for another, say, twenty years -- the fact that it's been around this long tells you that it's built well. (I know I'm invoking the Copernican principle here, which I've criticized before in reference to space exploration, but furniture isn't space exploration.) So this should make you be willing to pay more for it; if you are buying a table, the amount you are willing to pay for it is roughly how much you would pay to have a table for a year, times how many years you expect the table to last. (If you sell the table before it stops working, this still applies, because the price you can get for it is the result of the next buyer making this same calculation.) As time goes on, the table accumulates scratches and dings -- making it worth less per year -- but its life expectancy might go up. (I know this seems somewhat paradoxical; you might think that a table has, say, a 5% chance of breaking in any given year -- a table breaking might be a Poisson process. But I'm claiming that the fact that this table is still in one piece should lead you to believe that it's a Good Table. I'm saying this mostly because it seems to lead to results about the used-furniture market that feel right.) When a piece of furniture is new this first effect is the larger one; when it is old the second effect is the larger one. At the point at which the second effect is larger, that's when the price of the thing starts going up.
People also attach a certain value to "antique" furniture, which would also raise how much they're willing to pay for it; however, I don't think my table is anywhere near old enough for that to matter.
As it turns out, I live above a used furniture store. However, I suspect that the owner of that store doesn't like me much, so I don't think I'll ask him what he thinks about this.
25 July 2007
fracta;s. space-filling curves, and scientific revolutions
Mark Chu-Carroll at "Good Math, Bad Math" writes about space-filling curves. These are really counterintuitive things -- curves that eventually fill up, say, an entire square. There's a nice article about them at Wikipedia.
It won't surprise you to learn that these aren't "curves" in the sense that you might think of them; if I ask you to draw a "curve" you'll probably draw something that's what mathematicians would call "piecewise smooth". What this means, roughly, is that you can draw a piece of it without having any "kinks", then turn, then draw another such piece, and so on, doing this only a finite number of times. Space-filling curves don't have this property; they are made up of infinitely many such "pieces". Not surprisingly, they also have infinite length. These curves are made by an iterative process; in the case of the Hilbert curve:
The maximum distance halves and the length doubles with each step; as we iterate, each point in the square is arbitrarily close to a point on the curve (thus on the curve) and the curve is infinitely long.
Andrew Cook at "Statistical Modeling, Causal Inference, and Social Science" writes about the fractal nature of scientific revolutions, pointing to this earlier post of his. The idea is that science moves forward in what the evolutionary biologists call "punctuated equilibrium" -- at most points "not much" is getting done but occasionally big moves are made and in the end science gets done. (This is a bit unfair, though, because the scientists who are doing the "not much" are often collecting the sort of data that is exactly what the revolutinaries doing the paradigm shift will turn out to need.) If this is true, then we might say that all the science that will get done between year 0 ("now") and year 81 (which turns out to be 2088) gets done either in the first third of that period (between 0 and 27) or the last third (between 54 and 81). But then something similar happens on each of those periods -- all the science gets done between 0 and 9, 18 and 27, 54 and 63, or 80 and 81. If we repeat this, ad infinitium, we get that the set of times at which science is being done is the Cantor set, which has measure zero; furthermore the rate of scientific progress, when scientific progress is happening, must be infinite in order for any science to happen at all!
Of course, this is ridiculous. But it makes sense that science happens in bursts, and that each burst is made of smaller bursts, and so on; that there are periods of stasis between these bursts, but that some of these periods of stasis are more static than others; and so on. It's only the mathematician's insistence on taking the limit that makes this model not work. Furthermore, there's more than one kind of science, and it could happen that one discipline's burst is another discipline's period of stasis. And maybe a model like this is more likely to hold for the individual scientist (who has periods when they Get Things Done and periods when they don't) than for science as a whole.
But the periods when it looks like the scientist isn't doing anything might be essential. The subconsious is often doing work then. Perhaps there is something about the way our subconscious works -- in which bigger breakthroughs need longer fallow periods to precede them -- that leads to this fractal nature, with bursts upon bursts.
It won't surprise you to learn that these aren't "curves" in the sense that you might think of them; if I ask you to draw a "curve" you'll probably draw something that's what mathematicians would call "piecewise smooth". What this means, roughly, is that you can draw a piece of it without having any "kinks", then turn, then draw another such piece, and so on, doing this only a finite number of times. Space-filling curves don't have this property; they are made up of infinitely many such "pieces". Not surprisingly, they also have infinite length. These curves are made by an iterative process; in the case of the Hilbert curve:
- on the first iteration the curve has length 3/2 and each point is within √2/4 of the curve;
- on the second iteration the curve has length 15/4 and each point is within √2/8 of the curve;
- on the third iteration the curve has length 63/8 and each point is within √2/16 of the curve;
- on iteration n the curve has length 2n - 1/2n (it is made of 4n-1 segments of length 2-n) and each point is within √2/2n+1 of the curve.
The maximum distance halves and the length doubles with each step; as we iterate, each point in the square is arbitrarily close to a point on the curve (thus on the curve) and the curve is infinitely long.
Andrew Cook at "Statistical Modeling, Causal Inference, and Social Science" writes about the fractal nature of scientific revolutions, pointing to this earlier post of his. The idea is that science moves forward in what the evolutionary biologists call "punctuated equilibrium" -- at most points "not much" is getting done but occasionally big moves are made and in the end science gets done. (This is a bit unfair, though, because the scientists who are doing the "not much" are often collecting the sort of data that is exactly what the revolutinaries doing the paradigm shift will turn out to need.) If this is true, then we might say that all the science that will get done between year 0 ("now") and year 81 (which turns out to be 2088) gets done either in the first third of that period (between 0 and 27) or the last third (between 54 and 81). But then something similar happens on each of those periods -- all the science gets done between 0 and 9, 18 and 27, 54 and 63, or 80 and 81. If we repeat this, ad infinitium, we get that the set of times at which science is being done is the Cantor set, which has measure zero; furthermore the rate of scientific progress, when scientific progress is happening, must be infinite in order for any science to happen at all!
Of course, this is ridiculous. But it makes sense that science happens in bursts, and that each burst is made of smaller bursts, and so on; that there are periods of stasis between these bursts, but that some of these periods of stasis are more static than others; and so on. It's only the mathematician's insistence on taking the limit that makes this model not work. Furthermore, there's more than one kind of science, and it could happen that one discipline's burst is another discipline's period of stasis. And maybe a model like this is more likely to hold for the individual scientist (who has periods when they Get Things Done and periods when they don't) than for science as a whole.
But the periods when it looks like the scientist isn't doing anything might be essential. The subconsious is often doing work then. Perhaps there is something about the way our subconscious works -- in which bigger breakthroughs need longer fallow periods to precede them -- that leads to this fractal nature, with bursts upon bursts.
Propp's self-referential aptitude test
Check out the Self-referential aptitude test by Jim Propp. The test consists of twenty questions, each with five possible answers; the answer to each question depends in various ways on the other questions! There are various routes to deriving the answers. I'd say to see if you can solve it, but that takes quite some time!
I tried to come up with the answers by first assuming that the answers were a random string of letters from A to E and then changing the answers to various questions in order to agree with each other; unfortunately this didn't work, because the way in which the answers depend on each other is too complicated. I expected this process to converge to a solution fairly quickly, but it didn't.
There is a probabilistic solution, though, via genetic algorithms; Mark Van Dine writes about it. The idea is that we can tell how many of the questions are correct, and then "mutate" the answers a bit hoping that we gradually find the right answer. This took thousands of generations, though, which is why I couldn't do it by hand.
I give a (deterministic) solution below. First, I want to give a couple thoughts on it. Presenting a solution like this is tricky; one wants to minimize the number of "intermediate results" where we know, say, that a question can have one of two or three answers but we don't know which it is. These sorts of results are hard to hold in one's head. (My original path to the solution had more such intermediate results; I don't claim that this way of writing the solution is optimal.) Another thing one wants to reduce is contradictions that take a long time to derive; there are a couple points in the solution where I know that a question has two possible answers, so I assume one of them and seek a contradiction. If it takes a long time to find the contradiction, then one forgets which assumptions need to be "undone" when the contradiction is finally reached. These two principles that I've given here are probably good principles for the writing of mathematical proofs, as well. It's a good idea to minimize the number of things your reader has to remember at any given time, because remembering things is hard.
It looks to me like there's no trick to finding a solution, from looking at the other solutions given at Propp's web site; the other solutions given there seem about as complicated as mine.
Anyway, here we go.
First, the answer to #12 has to be (A): the question is "the number of questions whose answer is a consonant is: even, odd, a perfect square, prime, divisible by 5". So I assume on that there is only one "right" answer to this question; therfore we need a number between 0 and 20 which is exactly one of those things. Clearly any number is even or odd, but not both, so we need a number which isn't a square, prime, or divisible by 5. Thus the number of questions whose answer is a consonant is 6, 8, 12, 14, or 18, and the answer is (A), for "even".
I also let knowledge from the outside world enter; the answer to #20 is (E). #12 and #15 have the same answer, so #15 is (A). This means that #13 must be (D). And the answer to #5 must be (E), by the principle that questions have only one correct answer and (E) is always a correct answer to that question.
Next, #6 and #17 only refer to each other, so we can hope to work out the answers to them without reference to the other questions. First, neither one can have the answer (E), since "all of the above" doesn't make sense in this context. Thus neither can have answer (C), because if either 6 or 17 has answer (C) then the other must have answer (E). Reasoning similarly, neither can have answer (A). So both #6 and #17 have either (B) or (D) as the answer; they must be one of each but we can't determine the order.
Similarly, #10 and #16 only refer to each other; they're (A) and (D), respectively.
So far, then, we have the answer string
____E *___D _AD_A D*__E
where the two *s are one B and one D. Furthermore, we know that 6, 8, 12, 14 or 18 answers are consonants. So we move on to problem 8, which asks how many answers are vowels; this must be 2, 6, 8, 12, or 14. The only ones of these which are among the choices are (C) 6 or (E) 8. Looking at #7, then, we must have DC, DE, or EE for the answers to #7 and #8.
Next, we consider #3, #4, and #8. The numerical answers to the first two must sum to the numerical answer to the third. Furthermore, we've already fixed two answers of (E), so the answer to #3 is (C), (D), or (E). From #4 we know there are at least four A's. The answer to #8 is C or E, as we've already shown. Since there are at least two E's and at most eight vowels, there are at most six A's, so #4 is A, B, or C. The possible combinations of answers for these three questions are: CAC, CCE, DBE, EAE. If we take EAE here, then there are four E answers (#3, #5, #8, #20) already and no other E answers are possible; in particular #7 is D. Thus this are only six possibilities for the answers to #3, #4, #7, #8: CADC, CCDE, CCEE, DBDE, DBEE, EADE.
#9 can't be C (because #12 isn't C). And it can't be (E), because #13 is (D) and so the "next question with the same answer as this one" can't be #14.
#1 asks: "The first question whose answer is B, is question: (A) 1 (B) 2 (C) 3 (D) 4 (E) 5". It can't be (A) (because then the question would contradict itself) or (E) (because the answer to #5 isn't (B).) It can't be (C), because we just showed the answer to #3 isn't (B). So it must be either (B) or (D). Now, it seems that we have to make an arbitrary choice. We say #1 is (B). Thus #2 is also (B). So #7 and #8 must be the same; we see that they both have to be (E). Thus #5, #7, #8, and #20 are all (E); this is four (E)s, so #3 must be (E). But now there are five (E)'s, which isn't possible by examining the choices to #3. So our assumption that #1 had answer (B) is wrong; #1 must be (D).
Thus #2 isn't (B), meaning that #7 and #8 aren't the same. And #4 is (B). Thus the answers to #3, #4, #7, and #8 must be (D), (B), (D), (E) respectively, by looking at our previous list of possible answers to those four questions. The answers so far are
D_DBE *DE_A _AD_A D*__E
and we can at least see the solution taking shape.
Next, we look at #2 again. The answer to #10 is (A). Now, #11 can't be (A), because one of the questions preceding it has answer (B). Thus the answer to #2 isn't (E). Next, #2 can't be (C), because then #8 and #9 would be the same; but #8 is (E) and #9 can't be (E). So #2 is (A) or (D). But #2 can't be (D), because #2 refers to "the only two consecutive questions with identical answers" and #1 is already (D). So #2 is (A). Ths means #6 and #7 have the same answer, (D). Thus #17 is (B). The answer string so far is
DADBE DDE_A _AD_A DB__E.
There are five answers (A); we know this from #4. But there are less than five answers (C), since there are only five blanks left and #9 is not (C). So #18 isn't (B). Similarly, there are more than five answers (D) from the choices to #14, and exactly three answers (E) from #3. So #18 isn't (C) or (D) either; thus #18 is (A) or (E). But we've already accounted for the three answers (E) -- they're #5, #8, and #20. So #18 is (A). We now know there are the same number of answers (A) and (B), namely five of each. We've identified the five (A) answers, and two of the answers (B). The answer string so far is
DADBE DDE_A _AD_A DBA_E
and there are only two answers (B) and four blanks left. Three of the blanks have the answer (B).
Now, #11 must be (B) or (C), since there are 1 or 2 (B) among the first ten answers. Say it's (C); this means that there are two (B)s among the first ten answers, so #9 must be (B). But this means that #9 and #11 have the same answer, and we just said they didn't. So #11 is (B). This means that there is one (B) among the first ten answers, so #9 is not (B). (A) and (E) have already been ruled out for #9 (as the answers to the adjacent questions), and (C) was previously ruled out. So the answer to #9 is (D). The answer string is now
DADBE DDEDA BAD_A DBA_E
with three (B)s and two blanks; there are five answers (B), so the blanks must be (B), and we have
DADBE DDEDA BADBA DBABE
which is the answer. We know this is right because it spells the sentence "Dad bedded a bad bad babe".
I tried to come up with the answers by first assuming that the answers were a random string of letters from A to E and then changing the answers to various questions in order to agree with each other; unfortunately this didn't work, because the way in which the answers depend on each other is too complicated. I expected this process to converge to a solution fairly quickly, but it didn't.
There is a probabilistic solution, though, via genetic algorithms; Mark Van Dine writes about it. The idea is that we can tell how many of the questions are correct, and then "mutate" the answers a bit hoping that we gradually find the right answer. This took thousands of generations, though, which is why I couldn't do it by hand.
I give a (deterministic) solution below. First, I want to give a couple thoughts on it. Presenting a solution like this is tricky; one wants to minimize the number of "intermediate results" where we know, say, that a question can have one of two or three answers but we don't know which it is. These sorts of results are hard to hold in one's head. (My original path to the solution had more such intermediate results; I don't claim that this way of writing the solution is optimal.) Another thing one wants to reduce is contradictions that take a long time to derive; there are a couple points in the solution where I know that a question has two possible answers, so I assume one of them and seek a contradiction. If it takes a long time to find the contradiction, then one forgets which assumptions need to be "undone" when the contradiction is finally reached. These two principles that I've given here are probably good principles for the writing of mathematical proofs, as well. It's a good idea to minimize the number of things your reader has to remember at any given time, because remembering things is hard.
It looks to me like there's no trick to finding a solution, from looking at the other solutions given at Propp's web site; the other solutions given there seem about as complicated as mine.
Anyway, here we go.
First, the answer to #12 has to be (A): the question is "the number of questions whose answer is a consonant is: even, odd, a perfect square, prime, divisible by 5". So I assume on that there is only one "right" answer to this question; therfore we need a number between 0 and 20 which is exactly one of those things. Clearly any number is even or odd, but not both, so we need a number which isn't a square, prime, or divisible by 5. Thus the number of questions whose answer is a consonant is 6, 8, 12, 14, or 18, and the answer is (A), for "even".
I also let knowledge from the outside world enter; the answer to #20 is (E). #12 and #15 have the same answer, so #15 is (A). This means that #13 must be (D). And the answer to #5 must be (E), by the principle that questions have only one correct answer and (E) is always a correct answer to that question.
Next, #6 and #17 only refer to each other, so we can hope to work out the answers to them without reference to the other questions. First, neither one can have the answer (E), since "all of the above" doesn't make sense in this context. Thus neither can have answer (C), because if either 6 or 17 has answer (C) then the other must have answer (E). Reasoning similarly, neither can have answer (A). So both #6 and #17 have either (B) or (D) as the answer; they must be one of each but we can't determine the order.
Similarly, #10 and #16 only refer to each other; they're (A) and (D), respectively.
So far, then, we have the answer string
____E *___D _AD_A D*__E
where the two *s are one B and one D. Furthermore, we know that 6, 8, 12, 14 or 18 answers are consonants. So we move on to problem 8, which asks how many answers are vowels; this must be 2, 6, 8, 12, or 14. The only ones of these which are among the choices are (C) 6 or (E) 8. Looking at #7, then, we must have DC, DE, or EE for the answers to #7 and #8.
Next, we consider #3, #4, and #8. The numerical answers to the first two must sum to the numerical answer to the third. Furthermore, we've already fixed two answers of (E), so the answer to #3 is (C), (D), or (E). From #4 we know there are at least four A's. The answer to #8 is C or E, as we've already shown. Since there are at least two E's and at most eight vowels, there are at most six A's, so #4 is A, B, or C. The possible combinations of answers for these three questions are: CAC, CCE, DBE, EAE. If we take EAE here, then there are four E answers (#3, #5, #8, #20) already and no other E answers are possible; in particular #7 is D. Thus this are only six possibilities for the answers to #3, #4, #7, #8: CADC, CCDE, CCEE, DBDE, DBEE, EADE.
#9 can't be C (because #12 isn't C). And it can't be (E), because #13 is (D) and so the "next question with the same answer as this one" can't be #14.
#1 asks: "The first question whose answer is B, is question: (A) 1 (B) 2 (C) 3 (D) 4 (E) 5". It can't be (A) (because then the question would contradict itself) or (E) (because the answer to #5 isn't (B).) It can't be (C), because we just showed the answer to #3 isn't (B). So it must be either (B) or (D). Now, it seems that we have to make an arbitrary choice. We say #1 is (B). Thus #2 is also (B). So #7 and #8 must be the same; we see that they both have to be (E). Thus #5, #7, #8, and #20 are all (E); this is four (E)s, so #3 must be (E). But now there are five (E)'s, which isn't possible by examining the choices to #3. So our assumption that #1 had answer (B) is wrong; #1 must be (D).
Thus #2 isn't (B), meaning that #7 and #8 aren't the same. And #4 is (B). Thus the answers to #3, #4, #7, and #8 must be (D), (B), (D), (E) respectively, by looking at our previous list of possible answers to those four questions. The answers so far are
D_DBE *DE_A _AD_A D*__E
and we can at least see the solution taking shape.
Next, we look at #2 again. The answer to #10 is (A). Now, #11 can't be (A), because one of the questions preceding it has answer (B). Thus the answer to #2 isn't (E). Next, #2 can't be (C), because then #8 and #9 would be the same; but #8 is (E) and #9 can't be (E). So #2 is (A) or (D). But #2 can't be (D), because #2 refers to "the only two consecutive questions with identical answers" and #1 is already (D). So #2 is (A). Ths means #6 and #7 have the same answer, (D). Thus #17 is (B). The answer string so far is
DADBE DDE_A _AD_A DB__E.
There are five answers (A); we know this from #4. But there are less than five answers (C), since there are only five blanks left and #9 is not (C). So #18 isn't (B). Similarly, there are more than five answers (D) from the choices to #14, and exactly three answers (E) from #3. So #18 isn't (C) or (D) either; thus #18 is (A) or (E). But we've already accounted for the three answers (E) -- they're #5, #8, and #20. So #18 is (A). We now know there are the same number of answers (A) and (B), namely five of each. We've identified the five (A) answers, and two of the answers (B). The answer string so far is
DADBE DDE_A _AD_A DBA_E
and there are only two answers (B) and four blanks left. Three of the blanks have the answer (B).
Now, #11 must be (B) or (C), since there are 1 or 2 (B) among the first ten answers. Say it's (C); this means that there are two (B)s among the first ten answers, so #9 must be (B). But this means that #9 and #11 have the same answer, and we just said they didn't. So #11 is (B). This means that there is one (B) among the first ten answers, so #9 is not (B). (A) and (E) have already been ruled out for #9 (as the answers to the adjacent questions), and (C) was previously ruled out. So the answer to #9 is (D). The answer string is now
DADBE DDEDA BAD_A DBA_E
with three (B)s and two blanks; there are five answers (B), so the blanks must be (B), and we have
DADBE DDEDA BADBA DBABE
which is the answer. We know this is right because it spells the sentence "Dad bedded a bad bad babe".
24 July 2007
predicting baseball attendance
Back in July I wondered where baseball attendance numbers come from, and pointed to J. C. Bradbury's post about the Harry Potter effect on baseball attendance; he came up with a quick-and-dirty estimate of the number of people who didn't go to baseball games because they were reading the new Harry Potter book. This got me thinking -- how could we know how many people we expect to go to any given baseball game?
This paper by Brian Stoner gives the results of a regression analysis which forecasts the average attendance of a baseball team, over the course of a season, as a function of average ticket price, number of seats in the team's stadium, the previous year's attendance, the team payroll in the current year and number of wins in the previous year, the team's television market size, and whether or not the team has a new stadium; these turn out to explain most (98%) of the variation over the sample period. For the rest of this post, let's assume that we can predict a team's average attendance; we know it'll be, say, 32,000 per game. (Equivalently, we can predict the attendance for the entire season, since the average is just this figure divided by 81.)
But what I was wondering about is not average attendance, but attendance at a specific game, or equivalently how much the attendance is expected to be above or below the average. If you look at the attendance figures for a given team, their deviation from the average attendance depends on other figures which vary from day to day. First, there are factors that probably affect any outdoor activity, not just baseball:
I don't think these three variables could be treated independently; I'd probably prefer a day game in April (the nights can be very cold!) but a night game in July. And people probably care less about weekday versus weekend when school isn't in session.
There are then the factors that are unique to sports:
This paper by Brian Stoner gives the results of a regression analysis which forecasts the average attendance of a baseball team, over the course of a season, as a function of average ticket price, number of seats in the team's stadium, the previous year's attendance, the team payroll in the current year and number of wins in the previous year, the team's television market size, and whether or not the team has a new stadium; these turn out to explain most (98%) of the variation over the sample period. For the rest of this post, let's assume that we can predict a team's average attendance; we know it'll be, say, 32,000 per game. (Equivalently, we can predict the attendance for the entire season, since the average is just this figure divided by 81.)
But what I was wondering about is not average attendance, but attendance at a specific game, or equivalently how much the attendance is expected to be above or below the average. If you look at the attendance figures for a given team, their deviation from the average attendance depends on other figures which vary from day to day. First, there are factors that probably affect any outdoor activity, not just baseball:
- day of week -- weekend baseball is more popular than weekday baseball
- time of day -- day games and night games are different, although which is more popular probably depends on day of week and the season
- weather. Pre-purchased tickets probably depend on the average weather; for example, my family has a long-standing policy of not going to day games in June through August, because there's too much risk of the weather being unbearable. I suspect fans in colder-weather markets than Philadelphia avoid games in early April. (Philly can be cold in early April; I wore a winter coat to the Phillies' second game of the season this year.) After the many games canceled due to poor weather in April (including that Indians-Mariners series in Cleveland that got snowed out!),Gregory Goodrich, a meteorologist, examined the frequency of such "miserable baseball days". Walkup tickets, by contrast, probably depend quite heavily on the actual weather. Also, weather matters a lot less in a domed stadium; only in particularly miserable weather people won't even want to leave their homes.
I don't think these three variables could be treated independently; I'd probably prefer a day game in April (the nights can be very cold!) but a night game in July. And people probably care less about weekday versus weekend when school isn't in session.
There are then the factors that are unique to sports:
- quality of the visiting team. In general I suspect people want to see the home team play a good opponent more than a bad opponent. But at the same time people like to see the home team win. Perhaps people most want to see a visiting team with the same record as the home team, which maximizes the chances of a competitive game. (Note that I'm assuming that the quality of the home team is factored into the average attendance.)
- identity of the visiting team. When two teams in the same division play each other, I expect the attendance is higher than usual. Similarly, when teams have a long-standing rivalry more people come to games. This is distinct from the "quality of the visiting team" -- Phillies fans are more likely to come down to the ballpark to see the Mets than the Brewers or Dodgers, who have similar records as of this writing. Giants-Dodgers, Red Sox-Yankees, Cubs-Cardinals, etc. games probably sell more tickets than other intradivisional games (Giants-Diamondbacks, Red Sox-Orioles, Cubs-Astros, etc.)
- time of season. This is distinct from "weather" because while one might expect the same weather in May and September, the two are very different in the context of a baseball season; a game near the end of the season is usually seen to be "more exciting", at least if at least one of the two teams involved has the potential of making the playoffs.
- playoff importance. Games that are important in determining who ends up in the playoffs should have more attendance; for example, games where the two teams are first and second in their division or in a wild-card race.
where do baseball attendance numbers come from?
The Numbers Guy asks: Are Sports Teams Juicing Attendance Figures? The two most popular ways of measuring attendance are paid attendance and number of people who came through the turnstile.
I'm not sure what the best way to count attendance is. Is a team more interested in how many people were actually watching them, or how many tickets they sold? If you want to gauge, say, the level of public support for the team, the first of these might be more important. If you want to know how much money you've made, the second seems more important -- except that it doesn't actually matter how many tickets were sold, but rather how much those tickets were sold for. A lot of teams have adopted a policy of having different prices on different nights; 30,000 tickets sold at an average of $15 and 30,000 tickets sold at an average of $25 are clearly quite different things.
But I think you'll generally see paid-attendance numbers rather than turnstile numbers because the first one is bigger. And it probably doesn't matter which one teams report so long as they all do it the same way; I doubt that different teams in the same league are going to have substantially different no-show rates. (I'd be intrigued to learn I'm wrong, though.)
On a related note, my hometown baseball team -- the Philadelphia Phillies -- have been selling out a lot of games this season. You can see the attendance figures at baseball-reference.com. All three games of the series against the St. Louis Cardinals on July 13-15 were called "sellouts"; the attendance at those games was 43,838, 45,050, and 44,872 respectively. (See the AP article if you don't believe me o the first of those three games. Also, I was there and they said it was a sellout.) The Phillies have reported attendance as high as 45,537 (Sunday, June 17, against the Detroit Tigers). Apparently "sellout crowds" can differ as widely as 4%. This fact at first makes me suspect that the Phillies use the number of people to come through the gates, because they're not just magically making another 1700 seets appear some days that aren't there others. (I suppose it's possible they might be -- if so, let me know how!) The article on the game of July 13 calls it the "13th sellout of the season", meaning there have been fifteen sellouts so far. It turns out that the game of Friday, July 13 is the fifteenth-most-attended game this season; the attendance at the fifteen most attended games has been, in rank order:
45,537: 6/17
45,289: 7/1
45,165: 6/29 (2)
45,153: 6/2
45,129: 5/13
45,107: 4/29
45,102: 6/16
45,050: 7/14
45,026: 5/12
45,003: 6/30
44,872: 7/15
44,742: 4/2
44,336: 4/13
44,323: 6/28
43,838: 7/13
The Phillies claim a seating capacity of 43,500; for certain games they sell standing-room tickets, and I had believed the number of these was 500 per game. Yet fourteen times this season they've broken 44,000. Where are these people?
Finally, J. C. Bradbury at Sabernomics asks how many people didn't go to baseball games because of the new Harry Potter book; he concludes about six thousand. I'm curious how he predicted what the attendance would have been in the absence of the new book; given the small size of the effect I'm not sure if his numbers are at all valuable, and he admits as much.
I'm not sure what the best way to count attendance is. Is a team more interested in how many people were actually watching them, or how many tickets they sold? If you want to gauge, say, the level of public support for the team, the first of these might be more important. If you want to know how much money you've made, the second seems more important -- except that it doesn't actually matter how many tickets were sold, but rather how much those tickets were sold for. A lot of teams have adopted a policy of having different prices on different nights; 30,000 tickets sold at an average of $15 and 30,000 tickets sold at an average of $25 are clearly quite different things.
But I think you'll generally see paid-attendance numbers rather than turnstile numbers because the first one is bigger. And it probably doesn't matter which one teams report so long as they all do it the same way; I doubt that different teams in the same league are going to have substantially different no-show rates. (I'd be intrigued to learn I'm wrong, though.)
On a related note, my hometown baseball team -- the Philadelphia Phillies -- have been selling out a lot of games this season. You can see the attendance figures at baseball-reference.com. All three games of the series against the St. Louis Cardinals on July 13-15 were called "sellouts"; the attendance at those games was 43,838, 45,050, and 44,872 respectively. (See the AP article if you don't believe me o the first of those three games. Also, I was there and they said it was a sellout.) The Phillies have reported attendance as high as 45,537 (Sunday, June 17, against the Detroit Tigers). Apparently "sellout crowds" can differ as widely as 4%. This fact at first makes me suspect that the Phillies use the number of people to come through the gates, because they're not just magically making another 1700 seets appear some days that aren't there others. (I suppose it's possible they might be -- if so, let me know how!) The article on the game of July 13 calls it the "13th sellout of the season", meaning there have been fifteen sellouts so far. It turns out that the game of Friday, July 13 is the fifteenth-most-attended game this season; the attendance at the fifteen most attended games has been, in rank order:
45,537: 6/17
45,289: 7/1
45,165: 6/29 (2)
45,153: 6/2
45,129: 5/13
45,107: 4/29
45,102: 6/16
45,050: 7/14
45,026: 5/12
45,003: 6/30
44,872: 7/15
44,742: 4/2
44,336: 4/13
44,323: 6/28
43,838: 7/13
The Phillies claim a seating capacity of 43,500; for certain games they sell standing-room tickets, and I had believed the number of these was 500 per game. Yet fourteen times this season they've broken 44,000. Where are these people?
Finally, J. C. Bradbury at Sabernomics asks how many people didn't go to baseball games because of the new Harry Potter book; he concludes about six thousand. I'm curious how he predicted what the attendance would have been in the absence of the new book; given the small size of the effect I'm not sure if his numbers are at all valuable, and he admits as much.
housing rent maps
From Information Aesthetics (originally from boingboing): maps of San Francisco where different colors represent neighborhoods which are more or less expensive to rent in, from craigslist data. This is the work of Ethan Garner. The site appears to be overworked right now, so I can't actually check it out.
Anyway, I've been thinking that a map like this would be useful. However:
- as people have pointed out, the scale is relative, not absolute, so "yellow" on one map doesn't correspond to "yellow" on the other;
- I'd kind of like to see numbers. What I'd really want is a map that tells you "it'll probably cost you around $X per month to live here". The screenshot on boingboing is good enough that I can tell it's not there
- the implicit scale of .5 miles (the color for each point was determined by looking at places up to half a mile away) sounds too large, to me, at least for the city I know best (Philadelphia). There are plenty of neighborhoods I know which really shouldn't be lumped in with things which are half a mile away. Then again, there may be an issue of the sample size here; if you reduce that radius too much there's the possibility of wild fluctuations due to a few particularly expensive or particularly cheap apartments.
What I'd really like to see is something that separates out the various elements of the price of an apartment -- for example, how much more does a two-bedroom, two-bathroom apartment cost than a two-bedroom, one-bathroom? How much more does a place with central air cost than one without? How much is being a block closer to downtown, or to a subway station, worth? (This last one, I think, has probably been studied; I've heard a rule of thumb that people value their travel time at one-half of their hourly wage, so if you know where people living in a certain area tend to go, and how much money they make, you've got an answer.) I've developed various rules of thumb while looking for housing, but just when I think they work I find too many counterexamples. But I'm not sure if they're actually counterexamples or if the model is sound and some people just price weirdly, and I don't have enough data -- or enough statistical knowledge -- to go any further.
As for "why San Francisco?" I've seen a similar neighborhood map of San Francisco, which gets its data mostly from Craigslist housing posts, and I've never seen maps like this generated from actual data for any other city. Craigslist probably has better coverage of San Francisco than any other city. However, my instinct is that Craigslist is still a flawed source, because it overrepresents the sort of apartments in the sort of neighborhoods that young people who are on the Internet a lot like. I don't know of any better source, though.
I don't trust this "neighborhood project", though, for a very simple reason: craigslist housing posts come from landlords, and landlords will lie about properties that are near the border of two neighborhoods if one neighborhood is seen as significantly "better" than the other. (In Philadelphia, for example, I've seen places at 56th and Arch called "University City" when even the University City District doesn't get within three-quarters of a mile of there, and their definition is widely thought to be very generous.) My instinct is that the data coming from people looking for roommates is substantially different, because people are less likely to lie to people they have to live with than to people they're just going to take money from.
Also, neighborhood names change with time; Philadelphia's "Graduate Hospital" neighborhood didn't exist twenty years ago. (And it now seems like a really stupid name, because the hospital's not called that any more. Residents are probably likely to use the name that a neighborhood had when they moved in. Unfortunately it would be nearly impossible to create a map that said "this is where various neighborhood boundaries were in 1950; this is where they are today", because getting the data for where people thought neighborhoods were in 1950 would require combing through far too many old newspapers and such.
Anyway, I've been thinking that a map like this would be useful. However:
- as people have pointed out, the scale is relative, not absolute, so "yellow" on one map doesn't correspond to "yellow" on the other;
- I'd kind of like to see numbers. What I'd really want is a map that tells you "it'll probably cost you around $X per month to live here". The screenshot on boingboing is good enough that I can tell it's not there
- the implicit scale of .5 miles (the color for each point was determined by looking at places up to half a mile away) sounds too large, to me, at least for the city I know best (Philadelphia). There are plenty of neighborhoods I know which really shouldn't be lumped in with things which are half a mile away. Then again, there may be an issue of the sample size here; if you reduce that radius too much there's the possibility of wild fluctuations due to a few particularly expensive or particularly cheap apartments.
What I'd really like to see is something that separates out the various elements of the price of an apartment -- for example, how much more does a two-bedroom, two-bathroom apartment cost than a two-bedroom, one-bathroom? How much more does a place with central air cost than one without? How much is being a block closer to downtown, or to a subway station, worth? (This last one, I think, has probably been studied; I've heard a rule of thumb that people value their travel time at one-half of their hourly wage, so if you know where people living in a certain area tend to go, and how much money they make, you've got an answer.) I've developed various rules of thumb while looking for housing, but just when I think they work I find too many counterexamples. But I'm not sure if they're actually counterexamples or if the model is sound and some people just price weirdly, and I don't have enough data -- or enough statistical knowledge -- to go any further.
As for "why San Francisco?" I've seen a similar neighborhood map of San Francisco, which gets its data mostly from Craigslist housing posts, and I've never seen maps like this generated from actual data for any other city. Craigslist probably has better coverage of San Francisco than any other city. However, my instinct is that Craigslist is still a flawed source, because it overrepresents the sort of apartments in the sort of neighborhoods that young people who are on the Internet a lot like. I don't know of any better source, though.
I don't trust this "neighborhood project", though, for a very simple reason: craigslist housing posts come from landlords, and landlords will lie about properties that are near the border of two neighborhoods if one neighborhood is seen as significantly "better" than the other. (In Philadelphia, for example, I've seen places at 56th and Arch called "University City" when even the University City District doesn't get within three-quarters of a mile of there, and their definition is widely thought to be very generous.) My instinct is that the data coming from people looking for roommates is substantially different, because people are less likely to lie to people they have to live with than to people they're just going to take money from.
Also, neighborhood names change with time; Philadelphia's "Graduate Hospital" neighborhood didn't exist twenty years ago. (And it now seems like a really stupid name, because the hospital's not called that any more. Residents are probably likely to use the name that a neighborhood had when they moved in. Unfortunately it would be nearly impossible to create a map that said "this is where various neighborhood boundaries were in 1950; this is where they are today", because getting the data for where people thought neighborhoods were in 1950 would require combing through far too many old newspapers and such.
23 July 2007
life expectancies
The Numbers Behind Life Expectancy, from Carl Bialik's "The Numbers Guy" column at the Wall Street Journal.
Michael Moore said in Sicko that life expectancy is higher in Cuba than in the U.S.; CNN says it's the other way around; it turns out that so much computation goes into these calculations that there's probably a substantial amount of error. You might naively think that if life expectancy is, say, 77 years, that means that the average person born 77 years ago (in 1930) is just now getting around to dying. But the problem is that medical care isn't static, so this doesn't tell us how long people being born now should expect to live. So what's actually done is that one looks at how many people of age N in, say, 2005 survive to age N+1 (in 2006), and then these are chained together to tell us how many people would live to, say, age 80 if medical care remained as it is today and so the mortality rates remained constant. Basically, life expectancy is a moving target, because medical care changes substantially during a single person's life.
However, although the number "77" might not be that meaningful, I would guess that differences between those numbers for different populations which had been computed in the same way are valid to look at. A society where this number is 80 is probably healthier than one where it's 70.
But as many people point out, the Cuban statistics might not reflect what's actually going on in that country. It's difficult to know for sure.
Also, this means that the low life expectancies for countries in sub-Saharan Africa which have been affected by the AIDS epidemic are probably lower than one would naively expect; one hopes that the AIDS epidemic won't keep killing people at the same rates that it is now.
Michael Moore said in Sicko that life expectancy is higher in Cuba than in the U.S.; CNN says it's the other way around; it turns out that so much computation goes into these calculations that there's probably a substantial amount of error. You might naively think that if life expectancy is, say, 77 years, that means that the average person born 77 years ago (in 1930) is just now getting around to dying. But the problem is that medical care isn't static, so this doesn't tell us how long people being born now should expect to live. So what's actually done is that one looks at how many people of age N in, say, 2005 survive to age N+1 (in 2006), and then these are chained together to tell us how many people would live to, say, age 80 if medical care remained as it is today and so the mortality rates remained constant. Basically, life expectancy is a moving target, because medical care changes substantially during a single person's life.
However, although the number "77" might not be that meaningful, I would guess that differences between those numbers for different populations which had been computed in the same way are valid to look at. A society where this number is 80 is probably healthier than one where it's 70.
But as many people point out, the Cuban statistics might not reflect what's actually going on in that country. It's difficult to know for sure.
Also, this means that the low life expectancies for countries in sub-Saharan Africa which have been affected by the AIDS epidemic are probably lower than one would naively expect; one hopes that the AIDS epidemic won't keep killing people at the same rates that it is now.
proofs and deceptions by dissection
A friend of mine pointed me to this puzzle, which shows a triangle broken up into four pieces, and then the four pieces are rearranged in such a way that it appears that they now have total area one less than they did before. [edit, 5:43 pm: an anonymous commenter said that the link was broken. It's fixed now.]
What's really happening is that the "triangle" on the top and the "triangle" on the bottom are not triangles. However, it's nearly impossible to tell this by eye. The top "triangle" actually has its hypotenuse bent slightly inward; the bottom "triangle" has its hypotenuse bent slightly outward. If you superimposed the bottom "triangle" on the top one, there would be a long, thin sliver of area which would be contained in the bottom triangle but not in the top one; this turns out to have an area of exactly 1, making up for the "missing square".
One naturally expects the large "triangles" -- which appear to be right triangles with leg lengths 13 and 5 -- to be similar to the two small triangles. But the dark blue triangle has legs 8 and 3, and the green one has legs 5 and 2. It seems natural to compare the three right triangles by looking at the size of their smallest angle (since they're right triangles, this determines the shape uniquely); these are arctan(5/13), arctan(3/8), and arctan(2/5), or 21.04, 20.56, and 21.80 degrees, respectively. So the "bending" is on the order of one and a quarter degrees, which is hardly visible to the naked eye, especially since we naturally expect that a line that "looks" straight really is straight.
It's not a coincidence that the various numbers which are involved in this figure are Fibonacci numbers (2, 3, 5, 8, 13). The Fibonacci numbers have the property that the ratio between any two consecutive members of the sequence is very close to φ = (1+√5)/2, or about 1.618; the sequence approaches this limit quite quickly. For example, 13/8 = 1.625. The relevant ratios here are the ratios between Fibonacci numbers which are two positions apart (say, 5 and 13); these very quickly approach φ2 = 2.618...
I hadn't seen this particular example before, but there are other, similar puzzles which rely on these facts about the Fibonacci numbers; see Dissection Fallacy at MathWorld, in which an 8-by-8 checkerboard is broken up into pieces which appear to be rearranged to have area 63 or 65. The underlying idea is very much the same -- that we don't see the slight "imperfections of shape".
This isn't particularly special to the Fibonacci numbers, but can actually occur whenever the slopes of two lines are rational numbers which are close to each other; see, for example, the Curry Triangle at MathWorld, which relies on the fact that 2/5 and 3/7 are approximately equal. A bunch of Java applets illustrating such "dissection fallacies" can be found at Cut The Knot, and another illustration of this particular dissection can be found at Missing square puzzle on Wikipedia.
There's also a classic proof by dissection of the Pythagorean theorem (see about a third of the way down); essentially we can divide the two squares on the legs of a right triangle into pieces and rearrange them to form a square on the hypotenuse. Pedants will complain that you can't do this without proving that when you move regions around and rotate them their areas don't change -- and I suppose this is technically true -- but I think that these sorts of proofs give a much better intuition for why the Pythagorean theorem is true than, say, Euclid's proof.
What's really happening is that the "triangle" on the top and the "triangle" on the bottom are not triangles. However, it's nearly impossible to tell this by eye. The top "triangle" actually has its hypotenuse bent slightly inward; the bottom "triangle" has its hypotenuse bent slightly outward. If you superimposed the bottom "triangle" on the top one, there would be a long, thin sliver of area which would be contained in the bottom triangle but not in the top one; this turns out to have an area of exactly 1, making up for the "missing square".
One naturally expects the large "triangles" -- which appear to be right triangles with leg lengths 13 and 5 -- to be similar to the two small triangles. But the dark blue triangle has legs 8 and 3, and the green one has legs 5 and 2. It seems natural to compare the three right triangles by looking at the size of their smallest angle (since they're right triangles, this determines the shape uniquely); these are arctan(5/13), arctan(3/8), and arctan(2/5), or 21.04, 20.56, and 21.80 degrees, respectively. So the "bending" is on the order of one and a quarter degrees, which is hardly visible to the naked eye, especially since we naturally expect that a line that "looks" straight really is straight.
It's not a coincidence that the various numbers which are involved in this figure are Fibonacci numbers (2, 3, 5, 8, 13). The Fibonacci numbers have the property that the ratio between any two consecutive members of the sequence is very close to φ = (1+√5)/2, or about 1.618; the sequence approaches this limit quite quickly. For example, 13/8 = 1.625. The relevant ratios here are the ratios between Fibonacci numbers which are two positions apart (say, 5 and 13); these very quickly approach φ2 = 2.618...
I hadn't seen this particular example before, but there are other, similar puzzles which rely on these facts about the Fibonacci numbers; see Dissection Fallacy at MathWorld, in which an 8-by-8 checkerboard is broken up into pieces which appear to be rearranged to have area 63 or 65. The underlying idea is very much the same -- that we don't see the slight "imperfections of shape".
This isn't particularly special to the Fibonacci numbers, but can actually occur whenever the slopes of two lines are rational numbers which are close to each other; see, for example, the Curry Triangle at MathWorld, which relies on the fact that 2/5 and 3/7 are approximately equal. A bunch of Java applets illustrating such "dissection fallacies" can be found at Cut The Knot, and another illustration of this particular dissection can be found at Missing square puzzle on Wikipedia.
There's also a classic proof by dissection of the Pythagorean theorem (see about a third of the way down); essentially we can divide the two squares on the legs of a right triangle into pieces and rearrange them to form a square on the hypotenuse. Pedants will complain that you can't do this without proving that when you move regions around and rotate them their areas don't change -- and I suppose this is technically true -- but I think that these sorts of proofs give a much better intuition for why the Pythagorean theorem is true than, say, Euclid's proof.
five finger keyboards? I don't think so.
From Slashdot: Five Finger Keyboards from Trevor's Trinkets. Trevor suggests the possibility that one could have five-key "keyboards" that do everything that normal keyboards do, simply by having the possibility of pressing multiple keys at once and having every possible combination of keys mean something.
This seems like a bit of abuse of combinatorics to me. First, he points out that it would be possible to have 31 (25-1) possible "keys" by allowing any combination of "pressed" and "not pressed" for the five keys. This seems just barely feasible to me. The next suggestion is to take into account the order in which the keys are depressed, giving rise to 5 + 5*4 + 5*4*3 + 5*4*3*2 + 5*4*3*2*1 = 325 combinations. In general (although he doesn't point this out), for an n-key keyboard this would give rise to
n + n(n-1) + n(n-1)(n-2) + ... + n(n-1)(n-2)...1
= n! [1/(n-1)! + 1/(n-2)! + ... + 1/0!]
combinations, which is very nearly e(n!), since we have e = 1/0! + 1/1! + 1/2! + 1/3! + ... and the term in brackets above is this series with the very small terms omitted. (One can't always just chop off the "small terms" like this, but the terms decrease quickly enough that I can say this here.) In fact, it's e(n!)-1, rounded down to the nearest integer. (For the record, 120e = 326.19...)
Having key combinations that do something that are prefixes of other valid key combinations, though, seems like a Bad Idea. On your computer, for example, you might press Ctrl-C to copy something. C by itself does something. But Ctrl by itself doesn't do anything. That way if your fingers slip you haven't accidentally done Something Else. There's a notion of the prefix code (or "prefix-free code") which takes this into account. Huffman coding is perhaps the best example of this and are quite useful for compression of data, since they allow "letters" that occur more commonly to be encoded with shorter strings. (Morse code works similarly.)
Then he points out that if we take the order in which the keys are released into account, we'd have 15,685 combinations; this is again true, but requires far more finger dexterity than I think we can have the average person to have. Furthermore, this seems like an incredible tax on memory. (And remember that desigining for the "average" person is actually foolish, because about half of people are below average.)
In terms of memory, Trevor suggests that menus of some sort could appear which tell people which numbers lead to which keys, for example saying "1. a-i, 2. j-r" and so on; this seems to me like it would only work if there's some natural order to the characters to be entered. This means that Trevor's suggestion that this could be used for entering, say, Chinese characters wouldn't work so well; there's no natural order to those characters, as far as I know.
There's actually a fairly old convention for the input of alphabetic data that I rather like; I remember seeing it used for, say, getting stock quotes by phone. The convention was that one had A = 21, B = 22, C = 23, D = 31, E = 32, ..., Z = 94; that is, the usual telephonic pairing of letters with numbers, with a second digit required to uniquely specify the letter. (In terms of keystrokes per letter, this has very nearly the same efficiency as the code that a lot of present-day phones use, namely A = 2, B = 22, C = 222, D = 3, E = 33, ..., Z = 9999, about two keystrokes per letter.)
But as one of the people who has already commented at Trevor's blog pointed out, it's probably better to have voice-based interface than try to develop this any further. It may be theoretically possible to have very complicated input strings, but in designing a device for actual human beings to use we must take into account that people make mistakes.
This seems like a bit of abuse of combinatorics to me. First, he points out that it would be possible to have 31 (25-1) possible "keys" by allowing any combination of "pressed" and "not pressed" for the five keys. This seems just barely feasible to me. The next suggestion is to take into account the order in which the keys are depressed, giving rise to 5 + 5*4 + 5*4*3 + 5*4*3*2 + 5*4*3*2*1 = 325 combinations. In general (although he doesn't point this out), for an n-key keyboard this would give rise to
n + n(n-1) + n(n-1)(n-2) + ... + n(n-1)(n-2)...1
= n! [1/(n-1)! + 1/(n-2)! + ... + 1/0!]
combinations, which is very nearly e(n!), since we have e = 1/0! + 1/1! + 1/2! + 1/3! + ... and the term in brackets above is this series with the very small terms omitted. (One can't always just chop off the "small terms" like this, but the terms decrease quickly enough that I can say this here.) In fact, it's e(n!)-1, rounded down to the nearest integer. (For the record, 120e = 326.19...)
Having key combinations that do something that are prefixes of other valid key combinations, though, seems like a Bad Idea. On your computer, for example, you might press Ctrl-C to copy something. C by itself does something. But Ctrl by itself doesn't do anything. That way if your fingers slip you haven't accidentally done Something Else. There's a notion of the prefix code (or "prefix-free code") which takes this into account. Huffman coding is perhaps the best example of this and are quite useful for compression of data, since they allow "letters" that occur more commonly to be encoded with shorter strings. (Morse code works similarly.)
Then he points out that if we take the order in which the keys are released into account, we'd have 15,685 combinations; this is again true, but requires far more finger dexterity than I think we can have the average person to have. Furthermore, this seems like an incredible tax on memory. (And remember that desigining for the "average" person is actually foolish, because about half of people are below average.)
In terms of memory, Trevor suggests that menus of some sort could appear which tell people which numbers lead to which keys, for example saying "1. a-i, 2. j-r" and so on; this seems to me like it would only work if there's some natural order to the characters to be entered. This means that Trevor's suggestion that this could be used for entering, say, Chinese characters wouldn't work so well; there's no natural order to those characters, as far as I know.
There's actually a fairly old convention for the input of alphabetic data that I rather like; I remember seeing it used for, say, getting stock quotes by phone. The convention was that one had A = 21, B = 22, C = 23, D = 31, E = 32, ..., Z = 94; that is, the usual telephonic pairing of letters with numbers, with a second digit required to uniquely specify the letter. (In terms of keystrokes per letter, this has very nearly the same efficiency as the code that a lot of present-day phones use, namely A = 2, B = 22, C = 222, D = 3, E = 33, ..., Z = 9999, about two keystrokes per letter.)
But as one of the people who has already commented at Trevor's blog pointed out, it's probably better to have voice-based interface than try to develop this any further. It may be theoretically possible to have very complicated input strings, but in designing a device for actual human beings to use we must take into account that people make mistakes.
22 July 2007
payday loans
When Businesses Can Do Math, from Grey Matters -- interesting links about companies that can do math and use it to rip people off. Check out the fine print to this CashCall.com ad with Gary Coleman:
Yes, you read that right. The interest rate is almost ONE HUNDRED PERCENT. A person taking out such a loan will end up paying back a total of $9,095.10. Their commercial makes it sound like they're lending money because they "trust" people, but that's certainly not the case. I suspect that at least one of the two following things is true:
(Incidentally, I was looking for something to calculate the payments on a loan for me; the first google hit; I wanted to check the payments on the CashCall.com loan. It complains that 99.25 is not in the range from 0.01 to 99 and so I couldn't possibly have meant that interest rate. I did!
Here are CashCall.com's rates; it seems that in states where larger loans are offered, such as California, the interest rates on those larger loans are lower (as low as 21%). Also, for some reason they only offer loans in the amounts of $1,000, $2,000, $2,600, $5,075, $10,000, and $20,000; does anybody have any idea why?
As for the high interest rates, I've heard that small payday loans -- say, the type where someone borrows $100 and has to pay back $115 a couple weeks later -- have to have high interest rates because the cost of doing all the administrative work for the loan needs to be covered. But it seems a lot harder to believe this on loans such as those given by CashCall.com.
This reminds me of the Comcast "Service Protection Plan" I wrote about a few days ago, which I concluded was a ripoff. I was telling my father about this today, and he pointed out that "if they're trying to sell it to you, they expect to make money off of it". The difference is that Comcast was talking about a few dollars a month, whereas payday lenders are giving people a way to really trash their financial lives.
The APR for a typical loan of $2,600 is 99.25% with 42 monthly payments of $216.55
Yes, you read that right. The interest rate is almost ONE HUNDRED PERCENT. A person taking out such a loan will end up paying back a total of $9,095.10. Their commercial makes it sound like they're lending money because they "trust" people, but that's certainly not the case. I suspect that at least one of the two following things is true:
- The people taking out these loans have a very high default rate. And I mean very high; if they were giving these loans at 30% (which is a typical rate for people with credit cards who have made a ot of late payments) then the monthly payment would be $100.69 over 42 months (see this calculator), for a total amount paid of $4,228.98. So I'm inclined to assume that the percentage of people who pay their loans back is less than half of the percentage who pay their credit cards back.
- There's not much competition for this sort of loan; people actually see this commercial and make a phone call without bothering to shop around. This seems pretty reasonable, because the sort of people who would shop around for the best deal are probably less likely to get themselves in this sort of trouble in the first place. Therefore, the loan companies charge the highest interest rate they can legally get away with. In fact, it wouldn't surprise me to learn that in the state where these
(Incidentally, I was looking for something to calculate the payments on a loan for me; the first google hit; I wanted to check the payments on the CashCall.com loan. It complains that 99.25 is not in the range from 0.01 to 99 and so I couldn't possibly have meant that interest rate. I did!
Here are CashCall.com's rates; it seems that in states where larger loans are offered, such as California, the interest rates on those larger loans are lower (as low as 21%). Also, for some reason they only offer loans in the amounts of $1,000, $2,000, $2,600, $5,075, $10,000, and $20,000; does anybody have any idea why?
As for the high interest rates, I've heard that small payday loans -- say, the type where someone borrows $100 and has to pay back $115 a couple weeks later -- have to have high interest rates because the cost of doing all the administrative work for the loan needs to be covered. But it seems a lot harder to believe this on loans such as those given by CashCall.com.
This reminds me of the Comcast "Service Protection Plan" I wrote about a few days ago, which I concluded was a ripoff. I was telling my father about this today, and he pointed out that "if they're trying to sell it to you, they expect to make money off of it". The difference is that Comcast was talking about a few dollars a month, whereas payday lenders are giving people a way to really trash their financial lives.
The Phillies and the Qankees, part 2
About a month ago, I wrote a post about two semi-fictional sports teams called the Phillies and the Qankees. I asked: what is the probability that the Phillies win a best-of-seven series of games, given that they win each game with probability p? And how does this compare to the probability of them winning each individual game? It turned out that if we had p = 1/2 + ε, then the probability of them winning the series was about 1/2 + (35/16)ε (assuming ε was small). In general, the probability of winning a best-of-(2n-1) series (that is, a series that ends when the best team wins seven games) was about 1/2 + (1.11 n1/2) ε; I conjectured that the constant 1.11 was actually 2/√Ï€.
We can easily compute the average number of games that are actually played in a seven-game series. If we let p be the probability that the Phillies win any given game, and q be the probability that the Qankees (now do you see why I called them the Qankees?), then the number of games we expect to be played is just
4(q4 + p4) + 5(4q4p + 4p4q) + 6(10q4p2 + 10p4q2) + 7(20q4p3 + 20p4q3)
since q4 + p4 is the probability that the series goes 4 games, and so on. If, as in the previous post, we let p = 1/2 + ε and act as if ε2 = 0, it turns out that this is independent of ε and is 93/16, or 5.8125. (Of course, it's not actually independent of ε; for those who know what this means, if we expand the expected number of games as a power series in ε, there's no ε term but there is an ε2 term. (It turns out, though, that more World Series have gone the full seven games than would be expected, both because of which games are played at whose stadium and for reasons of baseball strategy.)
So in the "standard" system we have to play 5.8125 games, on average, to get an amplification of 35/16. Could we do better? Let's consider a playoff system that works as follows: the two teams play two best-of-three series. If the same team has won both series, they are declared the champions; if they've each won one series, then a third series is played to determine the champion. This is a best-of-three series of best-of-three series. The amplification is easy to find. If the Phillies have probability 1/2 + ε of winning each game, then we know from the previous post that they have probability 1/2 + (3/2)ε of winning a best-of-three series; thus they have probability 1/2 + (3/2)2ε of winning in this format. The amplification is a bit better than in a best-of-seven series, but only barely. It turns out, though, that on average you play more games in this format than in the best-of-seven format. On average, a best-of-three series consists of 2.5 games. (Half the time, the same team wins both games and it's over; the other half of the time, they split the first two games and the third has to be played.) So a best-of-three series of best-of-three series requires, on average, (2.5)2, or 6.25, games. It could take as few as four or as many as nine. (For evenly matched teams, though, the chances of going nine games are one-sixteenth -- a lot less than the probability of a best-of-seven series going its maximum length.)
At this point I started to wonder -- is it possible to design a system that gets substantially better amplification than something on the order of the square root of the number of games played? As I said before, it seems unlikely -- the whole situation reminds me of opinion polling, and if there were a way to do better than the square root there, I'd bet some pollster would have figured it out. Jordan Ellenberg suggested a solution in 2004, in which the World Series would end when one team is up 3-0, 4-1, 4-2, 5-3, or 5-4. I don't like this system for a best-of-seven series, because I was living in Boston when the 2004 American League Championship Series happened -- the Yankees (the real New York team, not the fictional Qankees) won the first three games but the Red Sox came back to win the last four. That could never have happened under this system, although it does open up possibilities for some other exciting series where a team repeatedly just barely avoids elimination and comes back to win.
How long is the average series now? The series can go three, five, six, eight, or nine games. The Phillies win in three with probability p3; the Qankees win in three with probability q3. For the Phillies to win in five, they have to lose the first, second, or third game and win the other four, with probability 3p4q; similarly, the Qankees win in five with probability 3q4p. For the Phillies to win in six, they must win the sixth game, and three of the first five games; there are ten ways to pick the games they win. But if they win the first three games, then the series would be over, so there are actually nine ways. Thus the Phillies win in six with probability 9p4q2; the Qankees win in six with probability 9q4p2. For the Phillies to win in eight, they must win the final two games, and three of the first six; but if the win the first three or lose the first three, it's over. There are thus 20-2 = 18 ways to pick the games they win, and their probability of winning in eight is 18p5q3; similarly, the Yankees win in eight with probability 18q5p3. Finally, we use a bit of trickery to determine the probability that the Phillies win in nine. This is some constant times p5q4. But if we let p=1/2, then we know the Phillies have probability 1/2 of winning the whole series; using this tells us the constant is 36.
The length of the average series turns out to be, then,
3(p3 + q3) + 5(3p4q< + 3q4p) + 6(9p4q2 + 9q4p2) + 8(18p5q3 + 18q5p3) + 9(36p5q4 + 36q5p4)
which, when p = 1/2, is 369/64, or 5.765625 -- very close to the length of the average World Series in our world. The Phillies' win probability is
p3 + 3p4q + 9p4q2 + 18p5q3 + 36p5q4
which, letting p = 1/2+ε where ε2 = 0, turns out to be 1/2 + (147/64)ε. 147/64 is about 2.30. It seems that in a series where about six games are played on average, the amplification will always be a bit greater than two.
But the purpose of the baseball playoffs isn't necessarily to find the objectively "best" team. In fact, a lot of people will argue that a good playoff team is different from a good regular-season team, because there are more off days in the playoffs than in the regular season. This means that once a team makes it to the playoffs they only need three starting pitchers, as opposed to the five they'd need during the regular season. The playoffs don't reward depth. If we wanted to find the objectively "best" team, we'd just award the championship to the team with the best regular-season record. The purpose of the playoffs is to provide entertainment. Since it seems like any playoff format with a "reasonable" average length is about equally good at finding the best team, it seems to me that baseball should stick with its current playoff system because it is familiar, and familiarity enables people to remember how things were in the past and compare those results to the present. Baseball is, of course, a game that is very aware of its own history.
We can easily compute the average number of games that are actually played in a seven-game series. If we let p be the probability that the Phillies win any given game, and q be the probability that the Qankees (now do you see why I called them the Qankees?), then the number of games we expect to be played is just
4(q4 + p4) + 5(4q4p + 4p4q) + 6(10q4p2 + 10p4q2) + 7(20q4p3 + 20p4q3)
since q4 + p4 is the probability that the series goes 4 games, and so on. If, as in the previous post, we let p = 1/2 + ε and act as if ε2 = 0, it turns out that this is independent of ε and is 93/16, or 5.8125. (Of course, it's not actually independent of ε; for those who know what this means, if we expand the expected number of games as a power series in ε, there's no ε term but there is an ε2 term. (It turns out, though, that more World Series have gone the full seven games than would be expected, both because of which games are played at whose stadium and for reasons of baseball strategy.)
So in the "standard" system we have to play 5.8125 games, on average, to get an amplification of 35/16. Could we do better? Let's consider a playoff system that works as follows: the two teams play two best-of-three series. If the same team has won both series, they are declared the champions; if they've each won one series, then a third series is played to determine the champion. This is a best-of-three series of best-of-three series. The amplification is easy to find. If the Phillies have probability 1/2 + ε of winning each game, then we know from the previous post that they have probability 1/2 + (3/2)ε of winning a best-of-three series; thus they have probability 1/2 + (3/2)2ε of winning in this format. The amplification is a bit better than in a best-of-seven series, but only barely. It turns out, though, that on average you play more games in this format than in the best-of-seven format. On average, a best-of-three series consists of 2.5 games. (Half the time, the same team wins both games and it's over; the other half of the time, they split the first two games and the third has to be played.) So a best-of-three series of best-of-three series requires, on average, (2.5)2, or 6.25, games. It could take as few as four or as many as nine. (For evenly matched teams, though, the chances of going nine games are one-sixteenth -- a lot less than the probability of a best-of-seven series going its maximum length.)
At this point I started to wonder -- is it possible to design a system that gets substantially better amplification than something on the order of the square root of the number of games played? As I said before, it seems unlikely -- the whole situation reminds me of opinion polling, and if there were a way to do better than the square root there, I'd bet some pollster would have figured it out. Jordan Ellenberg suggested a solution in 2004, in which the World Series would end when one team is up 3-0, 4-1, 4-2, 5-3, or 5-4. I don't like this system for a best-of-seven series, because I was living in Boston when the 2004 American League Championship Series happened -- the Yankees (the real New York team, not the fictional Qankees) won the first three games but the Red Sox came back to win the last four. That could never have happened under this system, although it does open up possibilities for some other exciting series where a team repeatedly just barely avoids elimination and comes back to win.
How long is the average series now? The series can go three, five, six, eight, or nine games. The Phillies win in three with probability p3; the Qankees win in three with probability q3. For the Phillies to win in five, they have to lose the first, second, or third game and win the other four, with probability 3p4q; similarly, the Qankees win in five with probability 3q4p. For the Phillies to win in six, they must win the sixth game, and three of the first five games; there are ten ways to pick the games they win. But if they win the first three games, then the series would be over, so there are actually nine ways. Thus the Phillies win in six with probability 9p4q2; the Qankees win in six with probability 9q4p2. For the Phillies to win in eight, they must win the final two games, and three of the first six; but if the win the first three or lose the first three, it's over. There are thus 20-2 = 18 ways to pick the games they win, and their probability of winning in eight is 18p5q3; similarly, the Yankees win in eight with probability 18q5p3. Finally, we use a bit of trickery to determine the probability that the Phillies win in nine. This is some constant times p5q4. But if we let p=1/2, then we know the Phillies have probability 1/2 of winning the whole series; using this tells us the constant is 36.
The length of the average series turns out to be, then,
3(p3 + q3) + 5(3p4q< + 3q4p) + 6(9p4q2 + 9q4p2) + 8(18p5q3 + 18q5p3) + 9(36p5q4 + 36q5p4)
which, when p = 1/2, is 369/64, or 5.765625 -- very close to the length of the average World Series in our world. The Phillies' win probability is
p3 + 3p4q + 9p4q2 + 18p5q3 + 36p5q4
which, letting p = 1/2+ε where ε2 = 0, turns out to be 1/2 + (147/64)ε. 147/64 is about 2.30. It seems that in a series where about six games are played on average, the amplification will always be a bit greater than two.
But the purpose of the baseball playoffs isn't necessarily to find the objectively "best" team. In fact, a lot of people will argue that a good playoff team is different from a good regular-season team, because there are more off days in the playoffs than in the regular season. This means that once a team makes it to the playoffs they only need three starting pitchers, as opposed to the five they'd need during the regular season. The playoffs don't reward depth. If we wanted to find the objectively "best" team, we'd just award the championship to the team with the best regular-season record. The purpose of the playoffs is to provide entertainment. Since it seems like any playoff format with a "reasonable" average length is about equally good at finding the best team, it seems to me that baseball should stick with its current playoff system because it is familiar, and familiarity enables people to remember how things were in the past and compare those results to the present. Baseball is, of course, a game that is very aware of its own history.
21 July 2007
why is there mental illness?
Mark Dominus at the Universe of Discourse makes an argument that homosexuality could be hereditary and yet still not ruled out by natural selection. Basically, the argument is that human sexuality is very complicated and isn't shaped by a single gene (which is patently obvious). We make the assumption that people having more than some number of "gay genes" turn out homosexual and people having less than that number turn out heterosexual. Then in a family where there are lots of people with lots of "gay genes", occasionally one of the kids turns out to be gay and doesn't reproduce, but then this person takes care of their nephews and nieces.
I'm not sure if I believe this, mainly because I know of no evidence that gay people actually pay more attention to their relatives who are not their children that straight people do. (Of course, just because I haven't heard it doesn't mean it's not there.)
Furthermore, on average people share half their genes with their children and one-quarter of them with a niece or nephew. So in order for this to work out in some sort of "expected value" framework, a gay person would have to be able to enhance the survival probability (or, more accurately, the expected number of children, or grandchildren) of their nephews and nieces by twice as much as they'd help their children, if they had them.
However, this could have an effect in times when "expected value" isn't what really matters -- when a family (and therefore a set of people with similar genes) are just barely clinging to life, very close to dying out. The logic then is that a straight person and their gay siblings can put all their eggs in one basket -- and then watch that basket very carefully.
Although I've never heard this sort of argument applied to homosexuality, I have heard it applied to various mental illnesses. There are people who believe that although, say, schizophrenia is obviously very harmful to the people who suffer from it, certain good qualities (say, high intelligence -- I don't remember if this is actually one of them!) tend to occur in the near relatives of schizophrenics. (Let me just say that I in no way am attempting to compare homosexuality and schizophrenia.)
Let's say, hypothetically, that there are ten genes, each of which occur in two variants called "red" and "blue", which cause schizophrenia. Let's say that each of these is "red" with probability p and "blue" with probability q = 1-p. Furthermore, a person which has zero or one of the "red" genes is "normal"; one who has two is of high intelligence, or "smart"; one who has three of more is schizophrenic. Assuming people mate at random, we are in a state of Hardy-Weinberg equilibrium. We compute:
What we see here is clear. When the frequency of red genes is low, most people are normal. When the frequency is moderate, we see a large minority of smart people and a small minority of schizophrenics. When the frequency is high, the schizophrenics begin to outnumber everybody else. Presumably, then, evolution would want (and here I commit the common sin of anthropomorphizing evolution) a moderate frequency of the "red" genes. As for how that is created, I think that assuming that high intelligence has survival value will do it; when the red genes are rare, "normal" people are the most common but the smart people will out-reproduce them, increasing the frequency of red genes; and when the red genes are common, schizophrenics are the most common but the smart people will out-reproduce them, decreasing the frequency of red genes. But even at the equilibrium point, not everyone will have exactly two red genes, which is what you need to be smart in this model. So there will still be variation.
Something similar actually goes on with the inheritance of sickle-cell anemia; having two copies of a certain allele gives people sickle-cell anemia, but having one copy of that allele confers resistance to malaria.
I'm not sure if I believe this, mainly because I know of no evidence that gay people actually pay more attention to their relatives who are not their children that straight people do. (Of course, just because I haven't heard it doesn't mean it's not there.)
Furthermore, on average people share half their genes with their children and one-quarter of them with a niece or nephew. So in order for this to work out in some sort of "expected value" framework, a gay person would have to be able to enhance the survival probability (or, more accurately, the expected number of children, or grandchildren) of their nephews and nieces by twice as much as they'd help their children, if they had them.
However, this could have an effect in times when "expected value" isn't what really matters -- when a family (and therefore a set of people with similar genes) are just barely clinging to life, very close to dying out. The logic then is that a straight person and their gay siblings can put all their eggs in one basket -- and then watch that basket very carefully.
Although I've never heard this sort of argument applied to homosexuality, I have heard it applied to various mental illnesses. There are people who believe that although, say, schizophrenia is obviously very harmful to the people who suffer from it, certain good qualities (say, high intelligence -- I don't remember if this is actually one of them!) tend to occur in the near relatives of schizophrenics. (Let me just say that I in no way am attempting to compare homosexuality and schizophrenia.)
Let's say, hypothetically, that there are ten genes, each of which occur in two variants called "red" and "blue", which cause schizophrenia. Let's say that each of these is "red" with probability p and "blue" with probability q = 1-p. Furthermore, a person which has zero or one of the "red" genes is "normal"; one who has two is of high intelligence, or "smart"; one who has three of more is schizophrenic. Assuming people mate at random, we are in a state of Hardy-Weinberg equilibrium. We compute:
- The probability of a person having zero or one "red" genes is P1 = q10 + 10q9p.
- The probability of a person having two "red" genes is P2 = 45q8p2.
- The probability of a person having three or more "red" genes is P3 = 1 - (P1 + P2). (It can be written out as the sum of the probabilities of having 3, 4, ..., 10 red genes, but it's easier to compute this way.)
| p | .02 | .05 | .10 | .15 | .20 | .25 | .30 | .35 |
| P2 | .01531 | .07463 | .19371 | .27590 | .30199 | .28157 | .23347 | .17565 |
| P3 | .00086 | .01550 | .07019 | .17980 | .32220 | .47441 | .61722 | .73839 |
What we see here is clear. When the frequency of red genes is low, most people are normal. When the frequency is moderate, we see a large minority of smart people and a small minority of schizophrenics. When the frequency is high, the schizophrenics begin to outnumber everybody else. Presumably, then, evolution would want (and here I commit the common sin of anthropomorphizing evolution) a moderate frequency of the "red" genes. As for how that is created, I think that assuming that high intelligence has survival value will do it; when the red genes are rare, "normal" people are the most common but the smart people will out-reproduce them, increasing the frequency of red genes; and when the red genes are common, schizophrenics are the most common but the smart people will out-reproduce them, decreasing the frequency of red genes. But even at the equilibrium point, not everyone will have exactly two red genes, which is what you need to be smart in this model. So there will still be variation.
Something similar actually goes on with the inheritance of sickle-cell anemia; having two copies of a certain allele gives people sickle-cell anemia, but having one copy of that allele confers resistance to malaria.
Comcast and dishonesty, part 1
I live in Philadelphia, home of Comcast. I can see Comcast's new tower in downtown Philly -- when completed, it'll be the "tallest buildling between New York and Chicago" -- from my living room window. An exercise for the reader: where do I get my Internet access from?
If you guessed Comcast, you're right.
Now, I recently had a bizarre customer service with them. The story goes as follows:
Saturday, noon. I'm sitting in my apartment. The Internet and the cable TV stop working, at essentially the same time. I call tech support, wait on hold, and so on. Now, whenever I talk to someone at Comcast's cable television division, they insist "oh, that's an Internet problem, that's not our problem." Hello? They come over the same wire. They go out at the same time. You expect me to believe this is two independent problems/ In the two years I've lived in the territory of Comcast or its predecessors, twice I've lost cable TV and Internet at the same time. I've never lost one but not the other.
Let's say there are three kinds of problems that can happen - one that makes just the TV go out, one that makes just the Internet go out, and one that makes both go out simultaneously. These are all "rare events" -- let's say each happens once a year -- and furthermore I'll say they're Poisson processes. (This basically means that future outages are not aware of past outages, which is technically called "memorylessness" or "independent increments". Now, let's say my cable and my Internet both went out within the last five minutes. What's the probability the same problem caused both?
The probability of losing my cable and Internet due to the same problem in any five-minute interval is five divided by the number of minutes in a year. Since I like RENT (the musical), I know that a year is 525,600 minutes, so this probability is one in 105,120; call it one in 100,000, since everything here is obscenely approximate anyway. But the probability that I lost them due to separate problems? It's the square of this, one in ten billion. So if I lived in this apartment for a hundred thousand years -- ten billion five-minute perioods -- then one hundred thousand times I will lose both TV and Internet due to the same problem. And one time, they'll just happen to go out within five minutes of each other, independently.
Conclusion? Even though I'm complaining about the Internet, it's your job to fix it. But you say "tell your landlord", because of course it's not your company's fault.
Saturday, 1pm. I call my landlord (a small, local property management company; their offices are closer than the closest mailbox, so I walk the rent over there instead of sticking it in the mail when it's due once a month.) The landlord says "it's not my problem", which is what I expected. then I get the tip which will haunt me for the next many days -- "I just let someone from Comcast into the basement of your building."
I run downstairs, chase down a guy in a Comcast pickup truck I see across the street. Someone -- him? one of his colleagues? I'm not sure -- had been there disconnecting people's cable because they weren't paying. I was paid up, but they'd made a mistake. The probabilistic moral here? This will happen less often if you live in a small building. (I've heard this about theft, as well -- the smaller the number of people who have keys to your building, the less likely your property is to be stolen.)
Another moral here is that Comcast ought to have a better system for telling which wire services which apartment, but that'll come later, when I talk about how their current system seems to work.
If you guessed Comcast, you're right.
Now, I recently had a bizarre customer service with them. The story goes as follows:
Saturday, noon. I'm sitting in my apartment. The Internet and the cable TV stop working, at essentially the same time. I call tech support, wait on hold, and so on. Now, whenever I talk to someone at Comcast's cable television division, they insist "oh, that's an Internet problem, that's not our problem." Hello? They come over the same wire. They go out at the same time. You expect me to believe this is two independent problems/ In the two years I've lived in the territory of Comcast or its predecessors, twice I've lost cable TV and Internet at the same time. I've never lost one but not the other.
Let's say there are three kinds of problems that can happen - one that makes just the TV go out, one that makes just the Internet go out, and one that makes both go out simultaneously. These are all "rare events" -- let's say each happens once a year -- and furthermore I'll say they're Poisson processes. (This basically means that future outages are not aware of past outages, which is technically called "memorylessness" or "independent increments". Now, let's say my cable and my Internet both went out within the last five minutes. What's the probability the same problem caused both?
The probability of losing my cable and Internet due to the same problem in any five-minute interval is five divided by the number of minutes in a year. Since I like RENT (the musical), I know that a year is 525,600 minutes, so this probability is one in 105,120; call it one in 100,000, since everything here is obscenely approximate anyway. But the probability that I lost them due to separate problems? It's the square of this, one in ten billion. So if I lived in this apartment for a hundred thousand years -- ten billion five-minute perioods -- then one hundred thousand times I will lose both TV and Internet due to the same problem. And one time, they'll just happen to go out within five minutes of each other, independently.
Conclusion? Even though I'm complaining about the Internet, it's your job to fix it. But you say "tell your landlord", because of course it's not your company's fault.
Saturday, 1pm. I call my landlord (a small, local property management company; their offices are closer than the closest mailbox, so I walk the rent over there instead of sticking it in the mail when it's due once a month.) The landlord says "it's not my problem", which is what I expected. then I get the tip which will haunt me for the next many days -- "I just let someone from Comcast into the basement of your building."
I run downstairs, chase down a guy in a Comcast pickup truck I see across the street. Someone -- him? one of his colleagues? I'm not sure -- had been there disconnecting people's cable because they weren't paying. I was paid up, but they'd made a mistake. The probabilistic moral here? This will happen less often if you live in a small building. (I've heard this about theft, as well -- the smaller the number of people who have keys to your building, the less likely your property is to be stolen.)
Another moral here is that Comcast ought to have a better system for telling which wire services which apartment, but that'll come later, when I talk about how their current system seems to work.
20 July 2007
Harry Potter and the pre-orders
As I wondered earlier: there are prediction markets in which you can bet on Harry Potter living. NewsFutures puts his probability of living at [redacted; click on the link if you want to know]. You can see a chart going back to March here. It's currently half past six in England; the answer will be publicly available in five and a half hours. Interestingly, the contract pays $100 (of virtual money) if Harry lives, $0 if Harry doesn't live, and $50 in "any other case".
Also, Amazon.com reveals that the Harry-est town in America is Falls Church, Virginia, as measured by the largest number of pre-orders per capita. (Only towns with more than 5,000 people were included.) The top twenty-two cities on this list are all within fifty-two miles of a major city. (Why 52? I was saying 50 at first, but Fredericksburg, VA was just outside the cutoff.) They are:
The Harry-est states in America is a different story; you'd expect from the first list that the states with lots of suburban population -- New Jersey or Virginia comes to mind, both states with no really large city within their borders but with one just outside -- would appear high up? Perhaps -- but the winner is actually the District of Columbia. (As a city, however, D.C. doesn't even make the top 100.) The six New England states are all high up -- Vermont is the highest at #2, Rhode Island the lowest at #16.
The moral of the story is that depending on how you sample you get very different results. Of course, the whole "Harry-est cities/states in America" thing is just a silly Amazon promotion.
Also, Amazon.com reveals that the Harry-est town in America is Falls Church, Virginia, as measured by the largest number of pre-orders per capita. (Only towns with more than 5,000 people were included.) The top twenty-two cities on this list are all within fifty-two miles of a major city. (Why 52? I was saying 50 at first, but Fredericksburg, VA was just outside the cutoff.) They are:
- Falls Church, VA (Washington, 10 miles)
- Gig Harbor, WA (Seattle 44)
- Fairfax, VA (Washington 21)
- Vienna, VA (Washington 16)
- Katy, TX (Houston 29)
- Media, PA (Philadelphia 22)
- Issaquah, WA (Seattle 17)
- Snohomish, WA (Seattle 31)
- Doylestown, PA (Philadelphia 40)
- Fairport, NY (Rochester 11)
- Woodinville, WA (Seattle 20)
- Princeton, NJ (Philadelphia 45; New York 51)
- Webster, NY (Rochester 15)
- West Chester, PA (Philadelphia 37)
- Williamsville, NY (Buffalo 11)
- Fredericksburg, VA (Washington 52)
- Port Orchard, WA (Seattle 22)
- Decatur, GA (Atlanta 6)
- Larchmont, NY (New York 27)
- Downingtown, PA (Philadelphia 39)
- Canton, GA (Atlanta 41)
- Woodstock, GA (Atlanta 31)
The Harry-est states in America is a different story; you'd expect from the first list that the states with lots of suburban population -- New Jersey or Virginia comes to mind, both states with no really large city within their borders but with one just outside -- would appear high up? Perhaps -- but the winner is actually the District of Columbia. (As a city, however, D.C. doesn't even make the top 100.) The six New England states are all high up -- Vermont is the highest at #2, Rhode Island the lowest at #16.
The moral of the story is that depending on how you sample you get very different results. Of course, the whole "Harry-est cities/states in America" thing is just a silly Amazon promotion.
pizza pie are square(d)
Geometry Saved Me Money, from Binary Dollar, via Grey Matters. Which is more: a twelve-inch pizza or two eight-inch pizzas?
The twelve-inch pizza, of course; it is more square inches of pizza. (I'm assuming all pizzas are equally thick.)
However, if you really like crust, the two eight-inch pizzas might actually be the better deal. One twelve-inch pizza contains 36Ï€ square inches of interior and 12Ï€ inches of crust; two eight-inch pizzas contain 32Ï€ square inches of interior and 16Ï€ inches of crust. So if you're willing to trade 4Ï€ square inches of interior for 4Ï€ inches of crust, take the smaller pizzas. That is, if you'd rather have a third of the crust of the pizza than a ninth of the interior, or if you'd prefer three crusts to one crustless slice.
I like the crust, so I might.
(This analysis assumes, of course, that the thickness of the crust is negligible, so that "an inch of crust" actually means something.)
The waitress in the restaurant where this question came up thought the customers would prefer the two eight-inch pizzas because it was more "slices of pizza". Maybe it's just me, but a "slice of pizza" is a meaningless unit, because it's not standard. I would have at least expected the argument that eight plus eight is more than twelve.
I suspect that pizza is the food that is most often "illogically" priced. I've seen, say, chicken wings sold at "10 for $5, 15 for $8" but you don't see that too often, because most people can do the math and realize that buying 15 is a bad deal. (Think of it this way: what if I want thirty? I can get three 10-packs for $15, or two 15-packs for $16.) But with pizza people will throw up their hands. Also, I have seen places where a larger size of pizza costs more per square inch than a smaller pizza (I can't find any right now); I was once told that this was because for whatever reason the large size was more inconvenient to make (it fit in the oven funny, for example). That at least seems like a plausible economic reason; it's clearly not the cost of the ingredients and almost certainly not the labor.
The twelve-inch pizza, of course; it is more square inches of pizza. (I'm assuming all pizzas are equally thick.)
However, if you really like crust, the two eight-inch pizzas might actually be the better deal. One twelve-inch pizza contains 36Ï€ square inches of interior and 12Ï€ inches of crust; two eight-inch pizzas contain 32Ï€ square inches of interior and 16Ï€ inches of crust. So if you're willing to trade 4Ï€ square inches of interior for 4Ï€ inches of crust, take the smaller pizzas. That is, if you'd rather have a third of the crust of the pizza than a ninth of the interior, or if you'd prefer three crusts to one crustless slice.
I like the crust, so I might.
(This analysis assumes, of course, that the thickness of the crust is negligible, so that "an inch of crust" actually means something.)
The waitress in the restaurant where this question came up thought the customers would prefer the two eight-inch pizzas because it was more "slices of pizza". Maybe it's just me, but a "slice of pizza" is a meaningless unit, because it's not standard. I would have at least expected the argument that eight plus eight is more than twelve.
I suspect that pizza is the food that is most often "illogically" priced. I've seen, say, chicken wings sold at "10 for $5, 15 for $8" but you don't see that too often, because most people can do the math and realize that buying 15 is a bad deal. (Think of it this way: what if I want thirty? I can get three 10-packs for $15, or two 15-packs for $16.) But with pizza people will throw up their hands. Also, I have seen places where a larger size of pizza costs more per square inch than a smaller pizza (I can't find any right now); I was once told that this was because for whatever reason the large size was more inconvenient to make (it fit in the oven funny, for example). That at least seems like a plausible economic reason; it's clearly not the cost of the ingredients and almost certainly not the labor.
gas prices fluctuating based on temperature
An Associated Press article claims that increased temperatures cost consumers money at the gas pump. (the link is to MSN Money Central.) The reason is not the usual seasonal fluctuations in gas prices, but rather the fact that gas is priced by volume, but the chemical reactions that fuel a car don't care about the volume of fuel you put in, but rather the mass. A United States liquid gallon has historically been defined as 231 cubic inches (although I suspect that now it, like most other units in our system, is defined as some exact multiple of some metric unit, in this case the cubic meter, liter, or cubic centimeter). But since gasoline, like other materials, expands as the temperature goes up. The article says that gas pumps "always dispense fuel as if it's 60 degrees". I assume this means that when it's warmer than 60 degrees, the "gallon" of gas you get from the pump is still one gallon by volume but it's slightly less mass -- about one percent less at 80 degrees than at 60 -- than it would be at 60 degrees and therefore your car doesn't go as far.
Consumer advocates are up in arms about this.
Personally, I'm a bit suspicious. For one thing, gas prices fluctuate wildly. If you are not blind, you can see this. (I mean this literally -- anyone in this country with eyes has some idea what gas costs, even if they don't buy any. I personally don't buy any -- I don't drive -- but I know that regular gas costs $2.96 a gallon right outside my apartment. Then again, I live across the street from a gas station.) I have no doubt that gas station owners are aware of this effect. Their margins on gasoline, depending on who you believe, are anywhere from 2% to 10%; this effect could seriously eat into their margins if they didn't know about this. The real money in running a gas station is from the attached convenience store.
I suspect that the news media over-reports on gas price fluctuations, though. "Gas prices went up this week" or "gas prices went down this week" gets people's attention; "gas prices stay the same this week" doesn't.
Second, a gallon is a unit of volume. In fact, I'm not totally sure what this calibration to 60 degrees even means. The case would seem a lot less silly to me if gas were being priced by the kilogram. (This leads me to wonder -- if gas were priced by the pound, would people complain that at higher altitudes gravity is weaker so a pound of gas has greater mass?)
Third, fitting the pumps to correct for temperature would be expensive -- a few thousand dollars a pump.
Fourth, 60 degrees was probably chosen because it's pretty close to the average annual temperature in much of the United States. So this might help people in the summer -- but it'll hurt them in the winter. (Of course, it is easy for me to say this, as both a non-driver and someone from a place where the average annual temperature is 54.3 degrees Fahrenheit.)
Still, it wouldn't surprise me if there are some people avoiding buying gas on hot days so that they can save a few cents a gallon. I'm not sure how viable such a strategy is, in part because the gas is stored in underground holding tanks and underground temperatures don't vary nearly as quickly as air temperatures. Second, what kind of savings are we talking here, a couple cents a gallon? Is it really worth stressing out over this to save thirty or forty cents on a tank of gas?
Consumer advocates are up in arms about this.
Personally, I'm a bit suspicious. For one thing, gas prices fluctuate wildly. If you are not blind, you can see this. (I mean this literally -- anyone in this country with eyes has some idea what gas costs, even if they don't buy any. I personally don't buy any -- I don't drive -- but I know that regular gas costs $2.96 a gallon right outside my apartment. Then again, I live across the street from a gas station.) I have no doubt that gas station owners are aware of this effect. Their margins on gasoline, depending on who you believe, are anywhere from 2% to 10%; this effect could seriously eat into their margins if they didn't know about this. The real money in running a gas station is from the attached convenience store.
I suspect that the news media over-reports on gas price fluctuations, though. "Gas prices went up this week" or "gas prices went down this week" gets people's attention; "gas prices stay the same this week" doesn't.
Second, a gallon is a unit of volume. In fact, I'm not totally sure what this calibration to 60 degrees even means. The case would seem a lot less silly to me if gas were being priced by the kilogram. (This leads me to wonder -- if gas were priced by the pound, would people complain that at higher altitudes gravity is weaker so a pound of gas has greater mass?)
Third, fitting the pumps to correct for temperature would be expensive -- a few thousand dollars a pump.
Fourth, 60 degrees was probably chosen because it's pretty close to the average annual temperature in much of the United States. So this might help people in the summer -- but it'll hurt them in the winter. (Of course, it is easy for me to say this, as both a non-driver and someone from a place where the average annual temperature is 54.3 degrees Fahrenheit.)
Still, it wouldn't surprise me if there are some people avoiding buying gas on hot days so that they can save a few cents a gallon. I'm not sure how viable such a strategy is, in part because the gas is stored in underground holding tanks and underground temperatures don't vary nearly as quickly as air temperatures. Second, what kind of savings are we talking here, a couple cents a gallon? Is it really worth stressing out over this to save thirty or forty cents on a tank of gas?
19 July 2007
more on that crazy Laffer curve
A few days ago I posted about some particularly egregious curve-fitting in the Wall Street Journal, in which corporate tax rates versus tax revenue in different countries was plotted, and a curve was fit to the data to make it look like we were on the right side of the Laffer curve. (Incidentally, people on the right economically believe we are on the right side of the Laffer curve, and therefore taxes should be cut; people on the left economically believe we are on the left side of the Laffer curve and therefore taxes should be raised.)
Someone else called this Best Curve-Fitting Ever (sarcastically, of course, if you take a look at the actual pictures); the comments are rather interesting, and do a better job of explaining the economic issues underlying this than I could.
Someone else called this Best Curve-Fitting Ever (sarcastically, of course, if you take a look at the actual pictures); the comments are rather interesting, and do a better job of explaining the economic issues underlying this than I could.
links: checkers, and the difference between mathematicians and statisticans
Checkers is a draw (from Scientific American); if this is true it's the most complicated game solved by computers to date.
(edit, 4:29 pm: the New York Times has picked up the checkers story. Jonathan Schaeffer, who led the research, says:
By this, it seems that he means that it doesn't "count" as a formal proof because basically what they did was go through all the possibilities that could occur in a game of checkers, without really having any insight into the game. I would still call it a formal proof, as it does establish beyond a doubt that there is a winning strategy for checkers (assuming that no errors slipped in) but I can understand Schaeffer's feeling that what he has done is not mathematics. I fear this sentence might give the wrong impression to laypeople, though.)
Math is like music, statistics is like literature: basically, math and music are both self-contained worlds, while practitioners of statistics and literature both benefit from having life experience. Thus the best mathematicians and musicians are younger than the best statisticians and novelists. (In the comments to this post, I claim that this differs among different branches of mathematics; the conventional wisdom says this is true but I know of no hard evidence.)
(edit, 4:29 pm: the New York Times has picked up the checkers story. Jonathan Schaeffer, who led the research, says:
“It’s a computational proof,” Dr. Schaeffer said. “It’s certainly not a formal mathematical proof.”
By this, it seems that he means that it doesn't "count" as a formal proof because basically what they did was go through all the possibilities that could occur in a game of checkers, without really having any insight into the game. I would still call it a formal proof, as it does establish beyond a doubt that there is a winning strategy for checkers (assuming that no errors slipped in) but I can understand Schaeffer's feeling that what he has done is not mathematics. I fear this sentence might give the wrong impression to laypeople, though.)
Math is like music, statistics is like literature: basically, math and music are both self-contained worlds, while practitioners of statistics and literature both benefit from having life experience. Thus the best mathematicians and musicians are younger than the best statisticians and novelists. (In the comments to this post, I claim that this differs among different branches of mathematics; the conventional wisdom says this is true but I know of no hard evidence.)
Alaskan glaciers, and the snows of Kilimanjaro
My parents recently got back from a two-week trip to Alaska.
While in Alaska, they saw some glaciers. I remembered that in An Inconvenient Truth, Al Gore mentioned how some glaciers in the Alps have retreated quite a bit within the historical record, showing pictures from the late 19th century and comparing them to modern-day photographs. It turned out that some of the glaciers there have receded within the memory of the people who live there, but others have actually advanced.
It seems to me that the only way to know for sure if global warming is causing glaciers to retreat is to analyze a large number of them for which there is a historical record. (Although I'm not a climatologist, the Alps seem like the best site for such a study simply because people have been living there for quite some time.) Anecdotal evidence just isn't valid here. However, it seems like there are better ways of tracking temperature than looking at glaciers (for example, just looking at actual temperature numbers!) and so the glaciers are more a convenient symbol than anything, because miles-long ice formations seem like something that humans just couldn't possibly melt! What one person knows is simply an inadequate sample, although it can make for a powerful story.
The news of the moment regarding Al Gore and global warming seems to be that the snows of Kilimanjaro are retreating, but that's not due to global warming but rather due to a comparatively dry weather period over the last century or so. The retreat began about a hundred years ago, whereas global warming didn't really become a big effect on climate until maybe forty or fifty years ago. The ice on Kilimanjaro is sublimating, not melting. Philip Mote of the University of Washington has said that Gore shouldn't use Kilimanjaro as a symbol for global warming, as he has in the past. He suggests that [t]here are dozens, if not hundreds, of photos of midlatitude glaciers you could show where there is absolutely no question that they are declining in response to the warming atmosphere,". I agree. Scientists are often blind to the power of good symbols in getting across their message to non-scientists.
Mathematicians, more so.
While in Alaska, they saw some glaciers. I remembered that in An Inconvenient Truth, Al Gore mentioned how some glaciers in the Alps have retreated quite a bit within the historical record, showing pictures from the late 19th century and comparing them to modern-day photographs. It turned out that some of the glaciers there have receded within the memory of the people who live there, but others have actually advanced.
It seems to me that the only way to know for sure if global warming is causing glaciers to retreat is to analyze a large number of them for which there is a historical record. (Although I'm not a climatologist, the Alps seem like the best site for such a study simply because people have been living there for quite some time.) Anecdotal evidence just isn't valid here. However, it seems like there are better ways of tracking temperature than looking at glaciers (for example, just looking at actual temperature numbers!) and so the glaciers are more a convenient symbol than anything, because miles-long ice formations seem like something that humans just couldn't possibly melt! What one person knows is simply an inadequate sample, although it can make for a powerful story.
The news of the moment regarding Al Gore and global warming seems to be that the snows of Kilimanjaro are retreating, but that's not due to global warming but rather due to a comparatively dry weather period over the last century or so. The retreat began about a hundred years ago, whereas global warming didn't really become a big effect on climate until maybe forty or fifty years ago. The ice on Kilimanjaro is sublimating, not melting. Philip Mote of the University of Washington has said that Gore shouldn't use Kilimanjaro as a symbol for global warming, as he has in the past. He suggests that [t]here are dozens, if not hundreds, of photos of midlatitude glaciers you could show where there is absolutely no question that they are declining in response to the warming atmosphere,". I agree. Scientists are often blind to the power of good symbols in getting across their message to non-scientists.
Mathematicians, more so.
Predicting names from initials
I am currently reading Paperboy: Confessions of a Future Engineer
by Henry Petroski, which I picked up a few days ago in a used bookstore; I'd previously read The Book on the Bookshelf
by the same author and enjoyed it.Petroski talks at one point (p. 114) about a boy named Frank, and he writes:
Frank's full name was Francis Xavier O'Connor, but he never told anybody his middle name. The X in Francis X. was officially the unknown, but nobody who was Catholic could imagine it being anything but Xavier.... Nine out of ten Catholics could guess correctly just from his initials, F. X. O'C., exactly what Frank's full name was"
This reminded me of something that I'd read before, something very similar with the initials F. X. O'B., the O'B standing for O'Brien. It appears that in Word Ways, the "Journal of Recreational Linguistics", in 1968, A. Ross Eckler wrote an article called "The Francis Xavier O'Brien Problem". It used to be available online, and I'd come across it before; I don't know whether Petroski read this or whether he independently came to the same conclusion. The chain of reasoning in both cases is as follows: O'B... hmm, that's got to be an Irish name; O'Brien is of course a very common Irish name. Good Irish Catholics name their kids after saints; Xavier's the only saint anyone's heard of whose name begins with X; the saint Xavier was actually Francis Xavier. If I remember correctly, Eckler claimed that someone initialed F. X. O'B. had a ninety percent chance of actually having the full name Francis Xavier O'Brien. If one Googles "Francis Xavier O'Brien", with the quotes, it seems that it's also common to name one's child "N Francis Xavier O'Brien", where N = Charles, John, James, etc. There are also 157 google hits for "Francis X. O'Brien"; replacing X in turn with the other letters of the alphabet, X is the fifth-most-common middle initial for Francis O'Briens, behind A, C, J, and W. (J is much more common as a first letter for a name than for a first letter of a non-name word.
This raises an interesting question: if you know someone's first and last name, what can you predict about their middle initial? I suspect some predictions can be made; I know quite a few people named either Mary Patricia Irishlastname or Patricia Mary Irishlastname, for example. I've also known two people named at least two people named Katherine Elizabeth Lastname (and both of their last names started with the same letter!). One thing that's clear is that people's names are not independent. If you don't believe me, just answer this question: do you seriously think you might meet someone whose full name is Bela Xavier Sedaris? (Bela Bollobas' Random Graphs and David Sedaris' Naked sit within arms' reach as I write this post.) Certain names correlate with certain ethnic groups, which in turn correlate with certain other names. It's not clear to me immediately whether just knowing someone's first and last initials would give information about their middle initial, but I don't see why it shouldn't.
18 July 2007
get your random numbers here!
From slashdot: a true random number generator goes online. It's called the quantum random bit generating service.
Note that these are classical bits it's generating -- things that are 0 or 1 -- and not quantum bits, which might not be totally obvious from the name. More information is available at the QRBG page. It appears that you can either get your random numbers online or get them from a little box you plug into the computer via a USB port.
The random number generators included in various programming languages are in fact "psuedorandom" numbers -- basically what this means is that they look random, but they are actually generated by a deterministic process. Often this process starts with some "seed" number and repeatedly performs some operation on it to spit out a sequence of random numbers. The random number generators used by modern technology pass known tests of randomness but there is always a possibility that there is some pattern that remains undiscovered. There are some cryptographic techniques that depend on numbers being random.
The technical paper about all this, "Quantum Random Number Generator" by M. Stipcevic and B. Medved Rogina, can be found at the research group's web site or at the arXiv. The way the algorithm works is roughly as follows: photons are emitted by a light source and pass through a beam splitters. The emission of photons can be assumed to be a Poisson process; what's important about this, for this process, is that the time between the emission of the second photon and the emission of the third photon doesn't depend on the time between the emission of the first photon and the second. We spit out "0" if the first of these is larger, "1" if it's smaller. Then we repeat for the 3rd, 4th, 5th photons (and thus the third and fourth intervals between photons); the 5th, 6th, 7th photons (and thus the fifth and sixth intervals between photons); and so on. Since the interval lengths are independent, this works. Of course, it might not work perfectly due to the limitations of the equipment, but the authors show that they need not worry.
Random.org does something similar; their random numbers come from "atmospheric noise". They're a little short on the technical details but you might find something of interest there. There's also lavarnd, which used to give random numbers that were derived from the input of a camera watching a lava lamp, but doesn't work that way any more.
Note that these are classical bits it's generating -- things that are 0 or 1 -- and not quantum bits, which might not be totally obvious from the name. More information is available at the QRBG page. It appears that you can either get your random numbers online or get them from a little box you plug into the computer via a USB port.
The random number generators included in various programming languages are in fact "psuedorandom" numbers -- basically what this means is that they look random, but they are actually generated by a deterministic process. Often this process starts with some "seed" number and repeatedly performs some operation on it to spit out a sequence of random numbers. The random number generators used by modern technology pass known tests of randomness but there is always a possibility that there is some pattern that remains undiscovered. There are some cryptographic techniques that depend on numbers being random.
The technical paper about all this, "Quantum Random Number Generator" by M. Stipcevic and B. Medved Rogina, can be found at the research group's web site or at the arXiv. The way the algorithm works is roughly as follows: photons are emitted by a light source and pass through a beam splitters. The emission of photons can be assumed to be a Poisson process; what's important about this, for this process, is that the time between the emission of the second photon and the emission of the third photon doesn't depend on the time between the emission of the first photon and the second. We spit out "0" if the first of these is larger, "1" if it's smaller. Then we repeat for the 3rd, 4th, 5th photons (and thus the third and fourth intervals between photons); the 5th, 6th, 7th photons (and thus the fifth and sixth intervals between photons); and so on. Since the interval lengths are independent, this works. Of course, it might not work perfectly due to the limitations of the equipment, but the authors show that they need not worry.
Random.org does something similar; their random numbers come from "atmospheric noise". They're a little short on the technical details but you might find something of interest there. There's also lavarnd, which used to give random numbers that were derived from the input of a camera watching a lava lamp, but doesn't work that way any more.
what happens to The Boy Who Lived? (no spoilers.)
At MSNBC's "iPredict" feature, you can vote on whether you think Harry Potter will live. (No spoilers.) There realy hasn't been any significant fluctuation since the voting started in early June.
This leads me to believe that people aren't taking the spoilers that are out there seriously (although I haven't read them, and I do not wish to), because if people knew whether Harry Potter was going to live or die you'd see a trend in one direction or the other. Then again, I suspect that prediction markets which involve actual money changing hands are better at predicting things, because people have a bit more incentive to make a correct prediction. I am not going to go seeking such prediction markets because I don't want to know the answer, and I have enough faith in prediction markets that I suspect they may know!
For what it's worth, I think he'll live -- I just don't see J. K. Rowling killing off a character loved by millions (although then again, she did kill off Dumbledore...) -- but I think that at some point it'll look like Harry will die. I don't think that he'll die and then be resurrected on the third day, but it would be kind of interesting if that happened.
What might be interesting to see is the probability, at any given moment in the book, that Harry will live. I'm inspired by the graphs at fangraphs.com, which show the probability of each team winning a baseball game after each plate appearance. (I swear this isn't a baseball blog!) It's not entirely clear what this means, though. The baseball probabilities are computed by looking at a sample of how things have gone in past games; there is only one Harry Potter. Could you compare it to other books? Could someone halfway through the book think "hmm, in most of the books I've read where it's been like this, the protagonist dies, so things don't going to look good for Harry?" But of course this couldn't be made into a prediction market, because the whole book is released at a single moment.
But what if you had some sort of medium where the story is released in pieces? Prediction markets for plot details of television shows -- even if you didn't let people trade during the episode's airing (because many TV shows air at different times in different places) -- could be interesting.
This leads me to believe that people aren't taking the spoilers that are out there seriously (although I haven't read them, and I do not wish to), because if people knew whether Harry Potter was going to live or die you'd see a trend in one direction or the other. Then again, I suspect that prediction markets which involve actual money changing hands are better at predicting things, because people have a bit more incentive to make a correct prediction. I am not going to go seeking such prediction markets because I don't want to know the answer, and I have enough faith in prediction markets that I suspect they may know!
For what it's worth, I think he'll live -- I just don't see J. K. Rowling killing off a character loved by millions (although then again, she did kill off Dumbledore...) -- but I think that at some point it'll look like Harry will die. I don't think that he'll die and then be resurrected on the third day, but it would be kind of interesting if that happened.
What might be interesting to see is the probability, at any given moment in the book, that Harry will live. I'm inspired by the graphs at fangraphs.com, which show the probability of each team winning a baseball game after each plate appearance. (I swear this isn't a baseball blog!) It's not entirely clear what this means, though. The baseball probabilities are computed by looking at a sample of how things have gone in past games; there is only one Harry Potter. Could you compare it to other books? Could someone halfway through the book think "hmm, in most of the books I've read where it's been like this, the protagonist dies, so things don't going to look good for Harry?" But of course this couldn't be made into a prediction market, because the whole book is released at a single moment.
But what if you had some sort of medium where the story is released in pieces? Prediction markets for plot details of television shows -- even if you didn't let people trade during the episode's airing (because many TV shows air at different times in different places) -- could be interesting.
SEPTA fare hike "11 percent"?
Funding heads off SEPTA fare hike.
The public transit provider in Philadelphia, SEPTA, recently raised its fares; the article is saying that they won't have to raise fares again in September. In the media the fare increase was widely reported as an "11 percent" increase. And it basically was. People who hold a weekly pass for the buses and subways saw the price raised from $18.75 to $20.75; monthly pass holders, from $70 to $78. A one-day pass was raised from $5.50 to $6. (A rule was made that a one-day pass was actually only good for eight trips, though.) SEPTA also runs commuter trains from the suburbs. A typical fare increase there (zone 3, off-peak) was from $3.75 to $4.25; a weekly pass for such a commuter train went up from $34.50 to $39, a monthly from $126.50 to $142.50.
So far, all the fare increases I've mentioned were between 9% and 13%. So far, so good.
But how much of a fare increase will I, personally, see? Zero.
Why is this? SEPTA decided to eliminate the transfer. Currently, there are three ways one can pay for a single trip on SEPTA: pay a $2 "cash fare", use a token which costs $1.30 (but tokens are only available in quantities of more than one, and it can be astonishingly difficult to find someplace which sells tokens!), or use a transfer, which costs 60 cents but can only be used for the second or third vehicle in a trip that requires more than one vehicle.
The cash fare and the token will remain at $2 and $1.30, respectively. The transfer will be eliminated. Because of where I happen to live and where I happen to go on a regular basis, and the way SEPTA's routes are structured, I never need to transfer. (Also, I never figured out how to buy a transfer.)
SEPTA says only 6.8 percent of riders use transfers. This makes sense; under the old fare structure, if you needed to use a transfer for a round-trip five days a week, that came to $19 a week; compare $18.75 for a pass. These figures are quite similar, and I'm assuming that our hypothetical person never goes anywhere but to work. The people who get screwed are the people who ride less then daily but need a transfer when they do so; they'll have to pay $2.60 (twice $1.30) for their trip instead of $1.90, a 37% fare increase. And these are probably the people who can least afford it.
I would have preferred to see, say, the $2 cash fare raised to $2.20, the token raised to $1.45, and the transfer continuing to exist (at 65 or 70 cents). (I haven't done the math to know if this would actually bring SEPTA the same amount of money as what they're doing; I'm just applying the 11% figure across the board.)
The article says a hearing is being held to restore the transfer. I think that's the right thing. It's not the rider's fault that SEPTA doesn't have a single route that goes from where they are to where they want to be; designing a transit system such that everyone has a one-seat ride would be essentially impossible. A good transit system works because of these network effects -- two routes are more than twice as valuable as one, a hundred routes are more than twice as valuable as fifty. But only if you make it possible to use the system as a system.
The public transit provider in Philadelphia, SEPTA, recently raised its fares; the article is saying that they won't have to raise fares again in September. In the media the fare increase was widely reported as an "11 percent" increase. And it basically was. People who hold a weekly pass for the buses and subways saw the price raised from $18.75 to $20.75; monthly pass holders, from $70 to $78. A one-day pass was raised from $5.50 to $6. (A rule was made that a one-day pass was actually only good for eight trips, though.) SEPTA also runs commuter trains from the suburbs. A typical fare increase there (zone 3, off-peak) was from $3.75 to $4.25; a weekly pass for such a commuter train went up from $34.50 to $39, a monthly from $126.50 to $142.50.
So far, all the fare increases I've mentioned were between 9% and 13%. So far, so good.
But how much of a fare increase will I, personally, see? Zero.
Why is this? SEPTA decided to eliminate the transfer. Currently, there are three ways one can pay for a single trip on SEPTA: pay a $2 "cash fare", use a token which costs $1.30 (but tokens are only available in quantities of more than one, and it can be astonishingly difficult to find someplace which sells tokens!), or use a transfer, which costs 60 cents but can only be used for the second or third vehicle in a trip that requires more than one vehicle.
The cash fare and the token will remain at $2 and $1.30, respectively. The transfer will be eliminated. Because of where I happen to live and where I happen to go on a regular basis, and the way SEPTA's routes are structured, I never need to transfer. (Also, I never figured out how to buy a transfer.)
SEPTA says only 6.8 percent of riders use transfers. This makes sense; under the old fare structure, if you needed to use a transfer for a round-trip five days a week, that came to $19 a week; compare $18.75 for a pass. These figures are quite similar, and I'm assuming that our hypothetical person never goes anywhere but to work. The people who get screwed are the people who ride less then daily but need a transfer when they do so; they'll have to pay $2.60 (twice $1.30) for their trip instead of $1.90, a 37% fare increase. And these are probably the people who can least afford it.
I would have preferred to see, say, the $2 cash fare raised to $2.20, the token raised to $1.45, and the transfer continuing to exist (at 65 or 70 cents). (I haven't done the math to know if this would actually bring SEPTA the same amount of money as what they're doing; I'm just applying the 11% figure across the board.)
The article says a hearing is being held to restore the transfer. I think that's the right thing. It's not the rider's fault that SEPTA doesn't have a single route that goes from where they are to where they want to be; designing a transit system such that everyone has a one-seat ride would be essentially impossible. A good transit system works because of these network effects -- two routes are more than twice as valuable as one, a hundred routes are more than twice as valuable as fifty. But only if you make it possible to use the system as a system.
what is the shape of a Moebius strip?
Loopy Logic: Moebius Strip Riddle Solved at Last, from Seed Magazine; also from Scientific American.
Who hasn't heard of the Mobius strip? The Mobius strip is the thing you get if you take a strip of paper, twist one of its ends half a twist, and tape the two ends together. It has the incredibly counterintuitive property that it only has one "side" -- if you were an ant, say, walking along the surface of such a strip, you could walk and walk and walk and suddenly you'd be where you were before but on what you'd think of as the "other side".
The article linked to -- which seems to be derived from a press release as I've found it in a few other places -- says that:
This seems a bit surprising -- it's easy to write down an equation that parametrizes the Mobius surface as basically a thickened circle. If you go to nature.com's writeup of it, though, it begins to make sense. What it actually means is that if you build a model of the Mobius strip out of an actual piece of paper or some other foldable material, the actual shape it will take is difficult to predict. This is what Eugene Starostin and Gert van der Heijden have actually done.
The difference here, then, is that when I hear "Mobius strip" I think of some sort of Platonic object, floating there in space, which could be realized in any material and is not subject to the laws of physics but only the laws of geometry, whereas the vast majority of people picture a piece of paper which has been twisted.
By the way, you can read some quasi-mystical stuff about the Mobius strip at Mobius Products and Services. As far as I know the Mobius strip has nothing to do with the Mobius transformation other than that the same person came up with both of them. But the legitimate journalists are subject to this mysticalism too -- nature.com points out that the Moebius strip, when viewed from a certain angle, looks like ∞, the sign for infinity.
Who hasn't heard of the Mobius strip? The Mobius strip is the thing you get if you take a strip of paper, twist one of its ends half a twist, and tape the two ends together. It has the incredibly counterintuitive property that it only has one "side" -- if you were an ant, say, walking along the surface of such a strip, you could walk and walk and walk and suddenly you'd be where you were before but on what you'd think of as the "other side".
The article linked to -- which seems to be derived from a press release as I've found it in a few other places -- says that:
Since 1930, the Moebius strip has been a classic poser for experts in mechanics. The teaser is to resolve the strip algebraically--to explain its unusual shape in the form of an equation.
This seems a bit surprising -- it's easy to write down an equation that parametrizes the Mobius surface as basically a thickened circle. If you go to nature.com's writeup of it, though, it begins to make sense. What it actually means is that if you build a model of the Mobius strip out of an actual piece of paper or some other foldable material, the actual shape it will take is difficult to predict. This is what Eugene Starostin and Gert van der Heijden have actually done.
The difference here, then, is that when I hear "Mobius strip" I think of some sort of Platonic object, floating there in space, which could be realized in any material and is not subject to the laws of physics but only the laws of geometry, whereas the vast majority of people picture a piece of paper which has been twisted.
By the way, you can read some quasi-mystical stuff about the Mobius strip at Mobius Products and Services. As far as I know the Mobius strip has nothing to do with the Mobius transformation other than that the same person came up with both of them. But the legitimate journalists are subject to this mysticalism too -- nature.com points out that the Moebius strip, when viewed from a certain angle, looks like ∞, the sign for infinity.
17 July 2007
how often is a team at .500?
The Phillies' current record is 46 wins and 46 losses.
When I heard this, I thought "hmm, the Phillies have been at .500 quite often this season". Baseball-reference.com tells us that they have been 0-0 (yes, that counts!) 20-20, 21-21, 22-22, 23-23, 24-24, 26-26, 28-28, 29-29, 44-44, and 46-46 this season; that's eleven times. Is that a lot? (I remember first noticing that between the 40th and 48th games of this season; after they were 20-20 they lost, won, lost, won, lost, won, lost, and won, in that order.)
Given that the team is 46-46, how many times should we expect them to have had the same number of wins and losses? It's a lot easier to work this out, of course, if we replace "46" with some smaller number.
For example, say the team had won two games and lost two games. Then there are C(4,2) = 6 ways we can arrange their two wins and two losses: WWLL, WLWL, WLLW, LWWL, LWLW, LLWW. In the first and last of these, the team was 0-0 and 2-2 at various times; in all the others they were also 1-1 after two games. This seems kind of obtuse, but let's flip things around. In six of these possibilities (which are all equally likely, because we've assumed the team wins exactly half its games), they're 0-0 after 0 games. In four of them, they're 1-1 after 2 games. In six of them, they're 2-2 after 4 games. The expected number of times that the team is at .500? It's (6+4+6)/6, or 16/6.
Sixteen is a power of two.
If we try this again for a 3-3 team, there are C(6,3) = 20 ways we can arrange three wins and three losses; there are 20, 12, 12, and 20 ways to arrange them so that the team is at some point 0-0, 1-1, 2-2, and 3-3 repspectively. So the total number of times we expect them to be at .500? It's (20+12+12+20)/20, or 64/20.
Sixty-four is again a power of two. Hmm, this can't be a coincidence.
Let's try to find that sum in the numerator in general. If the team has n wins and n losses (so eventually I'll set n=46 to solve the original problem), then how many ways are there to arrange the wins and losses so that the team wins m of the first 2m games? Clearly this is C(2m,m) C(2(n-m), n-m); we first have to pick which of the first 2m games are the first m wins, then which of the remaining 2(n-m) wins are the n-m remaining wins. So what we want to find is the sum
C(0,0) C(2n,n) + C(2,1) C(2n-2, n-1) + ... + C(2n-2, n-1) C(2,1) + C(2n, n) C(0,0)
and I don't see how to do this directly. However, consider the (infinite) power series
1 + 2z + 6z2 + 20z3 + 70z4 + ...
where the coefficients are C(0,0) = 1, C(2,1) = 2, C(4,2) = 6, C(6,3) = 20, C(8,4) = 70, and so on. (This is called the generating function of this series; generating functions are a ridiculously powerful tool which I will only scratch the surface of here.) This turns out to be the Taylor series of the function (1-4z)-1/2 at z=0. Now, consider what happens if we multiply this power series by itself, so we have
(1 + 2z + 6z2 + 20z3 + 70z4 + ...)(
1 + 2z + 6z2 + 20z3 + 70z4 + ...)
= (1)(1) + [(2)(1) + (1)(2)]z + [(6)(1)+(2)(2)+(1)(6)] z2 + [(20)(1)+(6)(2)+(2)(6)+(1)(20)] z3 + ...
and the coefficient of zn is exactly the sum we want to find! But the power series multiplied by itself is just (1-4z)-1, so the coefficient of zn is 4n.
Finally, we conclude that if we work out the expected number of times at .500 for a team with n wins and n losses, it's 4n/C(2n,n). But it's well-known that C(2n,n) is approximately 4n/(Ï€n)1/2. So a team with n wins and n losses is expected to have been at .500 very nearly (Ï€n)1/2 times.
When n=46, this approximation gives 12.021. (The exact number 446/C(92,46) is, to three decimal places, 12.054.) The Phillies have been at .500 eleven times so far; this is actually less than the expectation, which surprised me. A team which is .500 at the end of the season is expected to have been at .500 sixteen times during the season. For the Phillies, though, since they're already at 46-46, that adjusts the estimate upward, to around twenty-two.
In general, though, one might not want to use the expectation of a random variable like this. It's possible that most teams which are .500 at the end of the season really hit that mark in the middle of the season, but a very few teams are .500 some ridiculously large number of times. However, the most times a team can be at .500 over a 162-game regular season is 82 (0-0, 1-1, 2-2, ..., 81-81), so the expectation probably is a decent guess. Also, the expectation is often a lot more accessible than more detailed information. It is in this case, because I haven't figured out how to get the whole distribution yet, so I don't know the probability, say, that a 46-46 team has been at .500 eleven or more times. That seems harder to figure out, and the best way to find that number would probably be via a simulation; getting an exact, analytic answer doesn't seem easy.
When I heard this, I thought "hmm, the Phillies have been at .500 quite often this season". Baseball-reference.com tells us that they have been 0-0 (yes, that counts!) 20-20, 21-21, 22-22, 23-23, 24-24, 26-26, 28-28, 29-29, 44-44, and 46-46 this season; that's eleven times. Is that a lot? (I remember first noticing that between the 40th and 48th games of this season; after they were 20-20 they lost, won, lost, won, lost, won, lost, and won, in that order.)
Given that the team is 46-46, how many times should we expect them to have had the same number of wins and losses? It's a lot easier to work this out, of course, if we replace "46" with some smaller number.
For example, say the team had won two games and lost two games. Then there are C(4,2) = 6 ways we can arrange their two wins and two losses: WWLL, WLWL, WLLW, LWWL, LWLW, LLWW. In the first and last of these, the team was 0-0 and 2-2 at various times; in all the others they were also 1-1 after two games. This seems kind of obtuse, but let's flip things around. In six of these possibilities (which are all equally likely, because we've assumed the team wins exactly half its games), they're 0-0 after 0 games. In four of them, they're 1-1 after 2 games. In six of them, they're 2-2 after 4 games. The expected number of times that the team is at .500? It's (6+4+6)/6, or 16/6.
Sixteen is a power of two.
If we try this again for a 3-3 team, there are C(6,3) = 20 ways we can arrange three wins and three losses; there are 20, 12, 12, and 20 ways to arrange them so that the team is at some point 0-0, 1-1, 2-2, and 3-3 repspectively. So the total number of times we expect them to be at .500? It's (20+12+12+20)/20, or 64/20.
Sixty-four is again a power of two. Hmm, this can't be a coincidence.
Let's try to find that sum in the numerator in general. If the team has n wins and n losses (so eventually I'll set n=46 to solve the original problem), then how many ways are there to arrange the wins and losses so that the team wins m of the first 2m games? Clearly this is C(2m,m) C(2(n-m), n-m); we first have to pick which of the first 2m games are the first m wins, then which of the remaining 2(n-m) wins are the n-m remaining wins. So what we want to find is the sum
C(0,0) C(2n,n) + C(2,1) C(2n-2, n-1) + ... + C(2n-2, n-1) C(2,1) + C(2n, n) C(0,0)
and I don't see how to do this directly. However, consider the (infinite) power series
1 + 2z + 6z2 + 20z3 + 70z4 + ...
where the coefficients are C(0,0) = 1, C(2,1) = 2, C(4,2) = 6, C(6,3) = 20, C(8,4) = 70, and so on. (This is called the generating function of this series; generating functions are a ridiculously powerful tool which I will only scratch the surface of here.) This turns out to be the Taylor series of the function (1-4z)-1/2 at z=0. Now, consider what happens if we multiply this power series by itself, so we have
(1 + 2z + 6z2 + 20z3 + 70z4 + ...)(
1 + 2z + 6z2 + 20z3 + 70z4 + ...)
= (1)(1) + [(2)(1) + (1)(2)]z + [(6)(1)+(2)(2)+(1)(6)] z2 + [(20)(1)+(6)(2)+(2)(6)+(1)(20)] z3 + ...
and the coefficient of zn is exactly the sum we want to find! But the power series multiplied by itself is just (1-4z)-1, so the coefficient of zn is 4n.
Finally, we conclude that if we work out the expected number of times at .500 for a team with n wins and n losses, it's 4n/C(2n,n). But it's well-known that C(2n,n) is approximately 4n/(Ï€n)1/2. So a team with n wins and n losses is expected to have been at .500 very nearly (Ï€n)1/2 times.
When n=46, this approximation gives 12.021. (The exact number 446/C(92,46) is, to three decimal places, 12.054.) The Phillies have been at .500 eleven times so far; this is actually less than the expectation, which surprised me. A team which is .500 at the end of the season is expected to have been at .500 sixteen times during the season. For the Phillies, though, since they're already at 46-46, that adjusts the estimate upward, to around twenty-two.
In general, though, one might not want to use the expectation of a random variable like this. It's possible that most teams which are .500 at the end of the season really hit that mark in the middle of the season, but a very few teams are .500 some ridiculously large number of times. However, the most times a team can be at .500 over a 162-game regular season is 82 (0-0, 1-1, 2-2, ..., 81-81), so the expectation probably is a decent guess. Also, the expectation is often a lot more accessible than more detailed information. It is in this case, because I haven't figured out how to get the whole distribution yet, so I don't know the probability, say, that a 46-46 team has been at .500 eleven or more times. That seems harder to figure out, and the best way to find that number would probably be via a simulation; getting an exact, analytic answer doesn't seem easy.
Comcast's "Service Protection Plan".
I got my cable bill today.
Enclosed with my cable bill was an ad for Comcast's "Service Protection Plan". Comcast's policy is apparently to charge for "wire-related service calls". The ad says the following:
So this is only a good buy if I expect to have a service call every 22.25/3.30 = 6.74 months (if I'm a video-only customer) or 32.25/3.30 = 9.77 months (if I only have Internet and/or digital voice through them and don't have their television service, which I suspect is quite rare); for those people who might incur both kinds of charges, the relevant quantity is somewhere in between.
In any case, nobody's wiring is that bad, is it, that it needs fixing more than once a year? And if it is, don't you have bigger things to worry about than your cable TV? Comcast is probably making huge piles of money off of this.
You might say that the reason for a customer to buy this is for "insurance", and that my expected value calculation is sort of silly because you're not protecting against the average but against the unusual. And that would be a valid point if, say, they were offering "for a low monthly fee of $330, you'll be covered for charges which are usually $2225 or $3225". These numbers are in the right ballpark for, say, car insurance. But I would hope that people have the good sense to save enough money that an unexpected expense of $22.25 isn't going to hurt them.
(The fine print says that this isn't available to "customers in a residential building with multiple apartments", which describes me.)
Enclosed with my cable bill was an ad for Comcast's "Service Protection Plan". Comcast's policy is apparently to charge for "wire-related service calls". The ad says the following:
"For a low monthly fee of $3.30, you'll be covered for all inside wire-related service calls. Without the plan, regular service call charges will apply. Current service call charges are $22.25 for a video-only service call and $32.25 for a High-Speed Internet or Digital Voice service call."
So this is only a good buy if I expect to have a service call every 22.25/3.30 = 6.74 months (if I'm a video-only customer) or 32.25/3.30 = 9.77 months (if I only have Internet and/or digital voice through them and don't have their television service, which I suspect is quite rare); for those people who might incur both kinds of charges, the relevant quantity is somewhere in between.
In any case, nobody's wiring is that bad, is it, that it needs fixing more than once a year? And if it is, don't you have bigger things to worry about than your cable TV? Comcast is probably making huge piles of money off of this.
You might say that the reason for a customer to buy this is for "insurance", and that my expected value calculation is sort of silly because you're not protecting against the average but against the unusual. And that would be a valid point if, say, they were offering "for a low monthly fee of $330, you'll be covered for charges which are usually $2225 or $3225". These numbers are in the right ballpark for, say, car insurance. But I would hope that people have the good sense to save enough money that an unexpected expense of $22.25 isn't going to hurt them.
(The fine print says that this isn't available to "customers in a residential building with multiple apartments", which describes me.)
more thoughts on calorie counting
Last night I wrote about commercials which "overmathematize" eating.
This morning, in the New York Times, I read the article Calorie Labels May Clarify Options, Not Actions. New York City apparently has a new law requiring calorie counts to be posted in certain restaurants, and other places are considering similar laws.
The argument that various experts are making is that a lot of restaurant food is worse for you than people think. I said yesterday that I believe the human body will naturally behave in such a way as to have people eating the amount they should. Of course it should! Millions of years of evolution can't be wrong.
But there weren't restaurants when evolution did most of its work. And as far as I know, there has been no evolutionary tendency away from eating everything that's put in front of us because we don't know when there will be food again. Different people have differing appetites, which makes me suspect that given enough time living in a society like the one we have right now where high-calorie food is readily available, we'll evolve to not want more than the amount of food we need to live. But that's only true given enough time. Evolution is slow.
So maybe here the posting of calorie counts is a good idea.
Maybe.
But then say that someone knows they "should" be eating 2000 calories a day, given their current weight, age, gender, and lifestyle, in order to maintain that current weight. And they get to a restaurant for lunch and they see the 400-calorie burger, the 300-calorie fries, and the 200-calorie soda. (I'm making these numbers deliberately low, by the way.) None of those numbers are that large compared to 2000, so they think it's okay. But those numbers add up to 900, which is nearly half of 2000. Does this person know they're eating nearly half the number of calories they need?
In short, I suspect that though well-intended, the usefulness of this will be thwarted by the fact that a substantial number of Americans can't do basic arithmetic on three-digit numbers. (This probably explains the reason why the Weight Watchers "points" scheme is so popular -- it lowers the size of the numbers people have to keep track of.)
I believe this also explains a lot of the current state of the housing market -- people signed up for loans without realizing that the advertised payment wasn't even going to cover the interest -- but that's a different story.
This morning, in the New York Times, I read the article Calorie Labels May Clarify Options, Not Actions. New York City apparently has a new law requiring calorie counts to be posted in certain restaurants, and other places are considering similar laws.
The argument that various experts are making is that a lot of restaurant food is worse for you than people think. I said yesterday that I believe the human body will naturally behave in such a way as to have people eating the amount they should. Of course it should! Millions of years of evolution can't be wrong.
But there weren't restaurants when evolution did most of its work. And as far as I know, there has been no evolutionary tendency away from eating everything that's put in front of us because we don't know when there will be food again. Different people have differing appetites, which makes me suspect that given enough time living in a society like the one we have right now where high-calorie food is readily available, we'll evolve to not want more than the amount of food we need to live. But that's only true given enough time. Evolution is slow.
So maybe here the posting of calorie counts is a good idea.
Maybe.
But then say that someone knows they "should" be eating 2000 calories a day, given their current weight, age, gender, and lifestyle, in order to maintain that current weight. And they get to a restaurant for lunch and they see the 400-calorie burger, the 300-calorie fries, and the 200-calorie soda. (I'm making these numbers deliberately low, by the way.) None of those numbers are that large compared to 2000, so they think it's okay. But those numbers add up to 900, which is nearly half of 2000. Does this person know they're eating nearly half the number of calories they need?
In short, I suspect that though well-intended, the usefulness of this will be thwarted by the fact that a substantial number of Americans can't do basic arithmetic on three-digit numbers. (This probably explains the reason why the Weight Watchers "points" scheme is so popular -- it lowers the size of the numbers people have to keep track of.)
I believe this also explains a lot of the current state of the housing market -- people signed up for loans without realizing that the advertised payment wasn't even going to cover the interest -- but that's a different story.
do we need to go to Mars now?
A Survival Imperative for Space Colonization, from today's (Tuesday's) New York Times.
There's something called the "Copernican principle" which states, basically, that if we know nothing about how long a given process has been going on, we ought to assume that there is nothing special about the current moment. Therefore, if we look at something that is currently going on, the probability is one-half that we are in the first half of the thing's lifetime, and therefore it'll keep going on for longer than it's already gone on. For example, if you didn't know anything about history, you should assume that the United States has an even chance of lasting another 227 years. The usual practice in statistics is to form a 95% confidence interval; that works out to be that we are not in the last 2.5% or the first 2.5% of the United States' lifetime, that is, there's a 95% chance that the U.S. will last between 227/39 years (a little less than six years) and 227*39 years (about nine millennia).
Or another 400 years, since the Jamestown colony in 1607? When did the United States start? That makes this sort of analysis tricky.
The Copernican principle's principal exponent is J. Richard Gott III, and a long list of times when the principle works is given here. I'm a bit skeptical, because throughout the paper he predicts 95% confidence intervals. And then he says "well, 90% of the time the results are within the 95% interval" -- or 100% of the time -- and claims that's good enough. It's hard for me to be convinced. I'd be more convinced if he made predictions about the distribution of the remaining lifetime, or even predictions of the form "the U.S. has a 50% chance of lasting between 227/3 and 227*3 years".
Anyway, Gott goes on to argue that because of this Copernican principle, we ought to start trying to develop a colony on Mars in the next fifty years or so, because the spaceflight program has a significant chance of going extinct soon. But I don't think this is so. Gott has tested his principle on things like how long world leaders stay in power, how long Broadway shows last, and so on -- these are things for which there is a large sample. We have no idea how long space programs tend to last, because there have only been at most a handful in human history. Furthermore, I would guess that we are near the beginning of that part of human history when we are capable of spaceflight, given that we don't just wipe ourselves out as a species -- it seems unlikely that we'll forget how to go into space, given the distributed nature of information these days. (Then again, I suppose the people in charge of the Library of Alexandria would have said the same thing.)
I agree with the idea that if we want to perpetuate the human species into the far future, then we should attempt to colonize other planets, especially since we seem to be ruining the planet we have. But the probabilistic logic underlying this is hardly ironclad. Maybe it's reasonable to say that spaceflight will continue to exist as long as the human species itself does.
There's something called the "Copernican principle" which states, basically, that if we know nothing about how long a given process has been going on, we ought to assume that there is nothing special about the current moment. Therefore, if we look at something that is currently going on, the probability is one-half that we are in the first half of the thing's lifetime, and therefore it'll keep going on for longer than it's already gone on. For example, if you didn't know anything about history, you should assume that the United States has an even chance of lasting another 227 years. The usual practice in statistics is to form a 95% confidence interval; that works out to be that we are not in the last 2.5% or the first 2.5% of the United States' lifetime, that is, there's a 95% chance that the U.S. will last between 227/39 years (a little less than six years) and 227*39 years (about nine millennia).
Or another 400 years, since the Jamestown colony in 1607? When did the United States start? That makes this sort of analysis tricky.
The Copernican principle's principal exponent is J. Richard Gott III, and a long list of times when the principle works is given here. I'm a bit skeptical, because throughout the paper he predicts 95% confidence intervals. And then he says "well, 90% of the time the results are within the 95% interval" -- or 100% of the time -- and claims that's good enough. It's hard for me to be convinced. I'd be more convinced if he made predictions about the distribution of the remaining lifetime, or even predictions of the form "the U.S. has a 50% chance of lasting between 227/3 and 227*3 years".
Anyway, Gott goes on to argue that because of this Copernican principle, we ought to start trying to develop a colony on Mars in the next fifty years or so, because the spaceflight program has a significant chance of going extinct soon. But I don't think this is so. Gott has tested his principle on things like how long world leaders stay in power, how long Broadway shows last, and so on -- these are things for which there is a large sample. We have no idea how long space programs tend to last, because there have only been at most a handful in human history. Furthermore, I would guess that we are near the beginning of that part of human history when we are capable of spaceflight, given that we don't just wipe ourselves out as a species -- it seems unlikely that we'll forget how to go into space, given the distributed nature of information these days. (Then again, I suppose the people in charge of the Library of Alexandria would have said the same thing.)
I agree with the idea that if we want to perpetuate the human species into the far future, then we should attempt to colonize other planets, especially since we seem to be ruining the planet we have. But the probabilistic logic underlying this is hardly ironclad. Maybe it's reasonable to say that spaceflight will continue to exist as long as the human species itself does.
16 July 2007
not everything needs to be counted
"One eight-ounce glass counts as almost twenty-five percent of your fruit and vegetable servings." -- from a commercial for Florida orange juice I just saw on television.
Something about this "counts as" construction bothers me. It sounds to me like they're saying "it's not really fruit" or something like that, like eating is some sort of game.
Similarly, there's a commercial for the cereal Special K that, if I remember correctly,has a couple of really skinny girls deciding not to skip breakfast; if you write it and kingdom living, among others, have complained that this commercial feeds our national obsession with being thin. A calorie count is given at some point (200, I think?) and they show a bowl of cereal which certainly has more than 200 calories. (Take a look at the serving size written on your cereal box sometime. It's probably a lot less than what you eat when you eat cereal. They also seem to imply that this little bowl of cereal will tide someone over until lunch, which just isn't going to happen.)
In general, I feel that the human body knows wat it wants to eat, and that counting calories is kind of a silly idea. I would call this overmathematization, and I think it's something that various parts of our society are succumbing to -- the fact that box office numbers are becoming a big part of news broadcasts, for example, when really only the people who work in the movie business should care about those. Not everything needs to have a number.
(edit, 7:23pm: Wow, google is fast! I wondered if "overmathematization" was a word people had used, so I googled it. There are nine hits. One of them is this entry, which I posted twenty-three minutes ago.)
Something about this "counts as" construction bothers me. It sounds to me like they're saying "it's not really fruit" or something like that, like eating is some sort of game.
Similarly, there's a commercial for the cereal Special K that, if I remember correctly,has a couple of really skinny girls deciding not to skip breakfast; if you write it and kingdom living, among others, have complained that this commercial feeds our national obsession with being thin. A calorie count is given at some point (200, I think?) and they show a bowl of cereal which certainly has more than 200 calories. (Take a look at the serving size written on your cereal box sometime. It's probably a lot less than what you eat when you eat cereal. They also seem to imply that this little bowl of cereal will tide someone over until lunch, which just isn't going to happen.)
In general, I feel that the human body knows wat it wants to eat, and that counting calories is kind of a silly idea. I would call this overmathematization, and I think it's something that various parts of our society are succumbing to -- the fact that box office numbers are becoming a big part of news broadcasts, for example, when really only the people who work in the movie business should care about those. Not everything needs to have a number.
(edit, 7:23pm: Wow, google is fast! I wondered if "overmathematization" was a word people had used, so I googled it. There are nine hits. One of them is this entry, which I posted twenty-three minutes ago.)
Al Gore and gerrymandering
Al Gore writes in his new book The Assault on Reason
(pp. 238-239), commenting on the fact that U. S. Senators serve for longer terms (six years) than U. S. Representatives (two years):"Until recently, it followed that House seats turned over more frequently than Senate seats.
In recent years, that juxtaposition has flipped. In the four elections of the new millennium, House members have been reelected an average of 96 percent of the time, compared with Senate members, who have been reelected only 85 percent of the time, according to Rhodes Cook, an expert on election statistics. It is now statistically easier for a House member to be reelected three times in succession than it is for a senator to be reelected once. As a result, the Founder's intention that House members would be on a shorter `leash" and more responsive to the public than the Senate, has been stood on its head."
This is in a chapter in which Gore argues that a lot of what our nation's Founding Fathers stood for has been changed. You might say that it doesn't matter which house is the one that turns over quicker -- but the two chambers are not symmetric. One could distinguish a world in which the Senate was the chamber with more turnover than one in which the House was.
The mathematical content here isn't particularly strong. We can compute that the probability of a House member staying in power for six years (three House terms) is (.96)3, which is 88%; this isn't all that much more than 85%. Still, though, the conventional wisdom is that the House turns over faster than the Senate, so it surprised me to read this.
Of course, the reason why House seats turn over faster than Senate seats is due to gerrymandering. Gerrymandering, for those who don't know, is the practice of devising Congressional districts which are shaped in such a way as to take advantage of how members of the two parties are distributed in space. A particularly egregious example is the second district of Arizona, which is basically two disjoint pieces connected by the Grand Canyon. Nobody lives in the Grand Canyon, because it's a giant hole in the ground. The reason for doing this, of course, is that the people at the eastern and western ends are in the same political party.
I've seen proposals that require a certain amount of "compactness" in a district, usually measured as something like the area of a district over the square of its perimeter; the isoperimetric inequality tells us that this is minimized for a circle, and it's very large for districts like the Arizona 2nd.
There are, in fact, mathematical algorithms to abolish gerrymandering. The one that I linked to is called the "shortest splitline" algorithm; it works by splitting states up into districts by drawing lines which are as short as possible to successively divide the population in half. (I'm simplifying a bit; what I said is only strictly true if the number of districts is a power of two.) This ensures that the districts are at least convex (intuitively, this means they don't have "dents") if the state is; many of them are triangles or quadrilaterals. Since this pays absolutely no attention to the way people tend to vote, it's seen as "impartial".
I'm not sure how I feel about this, though, just because people's settlement patterns don't fall along straight lines. My instinct is to propose a system that tries not to divide counties if at all possible; historically counties in the United States are of roughly uniform size and shape (especially in sparsely settled western states like Nebraska, but even densely populated eastern states like New Jersey or Massachusetts are fairly compact. Furthermore, county lines can't be redrawn nearly as casually as congressional district lines can be. But there are plenty of counties that have a much larger population than a congressional district (roughly 700,000 at the moment), so this isn't perfect either.
Still, it's probably a lot better than the current system.
If you want to get your own experience with gerrymandering, there's something called the redistricting game which is floating around, which has been mentioned by the New York Times and Slashdot.
You're paying for the bottle, not the water
A couple weeks ago I came across a suspicious-sounding claim that New York City tap water costs 24 cents a gallon; the correct figure is 0.24 cents per gallon.
The New York Times gets it right today; in an article about how people drink more bottled water and less of just about everything else, they claim that
But with bottled water, you're really paying for the bottle, not the water.
On a related note, the Philadelphia Water Department is now giving out free bottles of tap water for events. The thinking is not financial, but environmental; they figure that if people are aware that tap water tastes good, then they won't use as much bottled water in the future. Bottled water is bad for the environment, because it's packaged in plastic containers and is often transported very long distances. Yesterday I saw someone buying FIJI water at the store on my corner. This comes from -- you guessed it -- Fiji, which is eight thousand miles away.
Personally, I buy bottled water for the convenience when I'm away from home but would never consider buying bottled water for my home. I've also been known to refill empty bottles with tap water, although there are rumors that that's unsafe.
The New York Times gets it right today; in an article about how people drink more bottled water and less of just about everything else, they claim that
As I said before, NYC water rates recently went up to 0.27 cents per gallon; assuming that "eight glasses" means "eight [measuring] cups", or half a gallon, this is correct. They then go on to point out that buying bottled water at the same rate would cost about $1,400 yearly. If you work it out, that's $4 a day, which values a 16-ounce bottle of bottled water at $1; that seems about right if you buy them individually, not in cases.
THOSE eight daily glasses of water you’re supposed to drink for good health? They will cost you $0.00135 — about 49 cents a year — if you take it from a New York City tap.
But with bottled water, you're really paying for the bottle, not the water.
On a related note, the Philadelphia Water Department is now giving out free bottles of tap water for events. The thinking is not financial, but environmental; they figure that if people are aware that tap water tastes good, then they won't use as much bottled water in the future. Bottled water is bad for the environment, because it's packaged in plastic containers and is often transported very long distances. Yesterday I saw someone buying FIJI water at the store on my corner. This comes from -- you guessed it -- Fiji, which is eight thousand miles away.
Personally, I buy bottled water for the convenience when I'm away from home but would never consider buying bottled water for my home. I've also been known to refill empty bottles with tap water, although there are rumors that that's unsafe.
15 July 2007
it's just a phone!
One-in-three Americans want iPhone -- or so Macworld claims. (Yeah, they're not biased at all.)
It's very thinly veiled advertising; in two clicks from that article I found the original press release from the research company. The "article" says:
I'm not sure if I trust this "online panel". For one thing, as a friend of mine pointed out, people who are online are more likely to want an iPhone. Second, the "article" goes on to say:
First of all, 8 plus 22 isn't 32, and even with rounding error that doesn't work out. More importantly, though, there are plenty of things that I've said I was going to buy in the future that I never ended up buying; if I put off buying something for long enough then I realize that I don't want it after all.
My friends seem like the sort of people that would want iPhones, and I think that not even one-third of them would want an iPhone -- although now I'll ask around.
Furthermore, you don't need to ask 39,000 people to get these sorts of results. But it sounds more impressive if you do. Most polls for, say, political campaigns have a sample size of about 1,000 and a 3% margin of error. With a well-chosen sample of 39,000 you should get a margin of error of about 0.5% (the margin of error scales as the inverse square root of the sample size), but I doubt this sample is well-chosen. I suspect that Lightspeed Research (the polling company) wasted their time asking this many people, because their sample is no good. They claim it is at their FAQ, and I believe their claim that the demographics of the panel are the same as the general population. However, they say that "A panel is comprised of people who have opted-in to share their views on products and services and recruited solely for market research purposes." The sort of people who are likely to be in such a panel are people who spend too much time online -- exactly the iPhone's target market!
The moral of this story: just because you asked a lot of people doesn't make you right.
It's very thinly veiled advertising; in two clicks from that article I found the original press release from the research company. The "article" says:
Lightspeed Research surveyed 39,000 people on its US online panel in the days following the launch of the device on 29 June - and the research findings are staggering.
I'm not sure if I trust this "online panel". For one thing, as a friend of mine pointed out, people who are online are more likely to want an iPhone. Second, the "article" goes on to say:
Thirty-two per cent of those surveyed who do not currently own an iPhone stated that they do intend to purchase one, with 8 per cent planning to purchase in the next three months and 22 per cent planning to purchase "some time in the future", the researchers said.
First of all, 8 plus 22 isn't 32, and even with rounding error that doesn't work out. More importantly, though, there are plenty of things that I've said I was going to buy in the future that I never ended up buying; if I put off buying something for long enough then I realize that I don't want it after all.
My friends seem like the sort of people that would want iPhones, and I think that not even one-third of them would want an iPhone -- although now I'll ask around.
Furthermore, you don't need to ask 39,000 people to get these sorts of results. But it sounds more impressive if you do. Most polls for, say, political campaigns have a sample size of about 1,000 and a 3% margin of error. With a well-chosen sample of 39,000 you should get a margin of error of about 0.5% (the margin of error scales as the inverse square root of the sample size), but I doubt this sample is well-chosen. I suspect that Lightspeed Research (the polling company) wasted their time asking this many people, because their sample is no good. They claim it is at their FAQ, and I believe their claim that the demographics of the panel are the same as the general population. However, they say that "A panel is comprised of people who have opted-in to share their views on products and services and recruited solely for market research purposes." The sort of people who are likely to be in such a panel are people who spend too much time online -- exactly the iPhone's target market!
The moral of this story: just because you asked a lot of people doesn't make you right.
infinity times two
Infinity, from the classic math television show "Square One".
I got this from meep, who points out: "A little ditty about infinity - and is good about showing infinity as a process, not a number." This is definitely true; although I don't want to transcribe the full lyrics, the Big Idea here is that infinity is not just a very large number. As the Hitchhiker's Guide to the Galaxy says (I don't know exactly where), "Infinity is just so big that, by comparison, bigness itself looks really titchy." In Chapter 24 of the first book, there's the following:
This makes quite a bit of sense; psychologically I don't think that we're built to understand the concept of infinity (or infinitesimals, for that matter), because we evolved to deal with medium-sized things. Mathematics, in a sense, is profoundly un-evolutionary, in that we've advanced it so far that it goes far beyond any intuition a "normal" human being might have. This probably explains why it's so hard for a lot of people to learn.
(One day I might write the Mathematical Hitchhiker's Guide to the Hitchhiker's Guide to the Galaxy, or something like that. There are a lot of math jokes in that series.)
Note: could you please let me know if you can see the embedded YouTube video in this post? I'm not sure if I did it correctly.
I got this from meep, who points out: "A little ditty about infinity - and is good about showing infinity as a process, not a number." This is definitely true; although I don't want to transcribe the full lyrics, the Big Idea here is that infinity is not just a very large number. As the Hitchhiker's Guide to the Galaxy says (I don't know exactly where), "Infinity is just so big that, by comparison, bigness itself looks really titchy." In Chapter 24 of the first book, there's the following:
The car shot forward straight into the circle of light, and suddenly Arthur had a fairly clear idea of what infinity looked like.
It wasn't infinity in fact. Infinity itself looks flat and uninteresting. Looking up into the night sky is looking into infinity — distance is incomprehensible and therefore meaningless. The chamber into which the aircar emerged was anything but infinite, it was just very very big, so that it gave the impression of infinity far better than infinity itself.
This makes quite a bit of sense; psychologically I don't think that we're built to understand the concept of infinity (or infinitesimals, for that matter), because we evolved to deal with medium-sized things. Mathematics, in a sense, is profoundly un-evolutionary, in that we've advanced it so far that it goes far beyond any intuition a "normal" human being might have. This probably explains why it's so hard for a lot of people to learn.
(One day I might write the Mathematical Hitchhiker's Guide to the Hitchhiker's Guide to the Galaxy, or something like that. There are a lot of math jokes in that series.)
Note: could you please let me know if you can see the embedded YouTube video in this post? I'm not sure if I did it correctly.
The Riemann hypothesis is probabilistic
Terence Tao, at his blog, describes a rather interesting-sounding interdisciplinary conference he recently attended. At this conference, he gave a talk about Structure and Randomness in the Prime Numbers, which he describes as a shortened version of a similar talk he gave at UCLA. You can see the slides and video of his UCLA talk here. The slides are not terribly useful, because Tao is a good speaker. Like a good speaker, he doesn't just read his slides; the slides give you an idea of what he's talking about but leave out perhaps 90% of the context. The video's good, but it's in RealPlayer format, and RealPlayer and Windows XP don't cooperate -- I got a Blue Screen of Death while watching.
Anyway, one of the ideas he mentions in this talk -- and this is something that I was not aware of before hearing this talk -- is that the Riemann hypothesis, which isequivalent to a certain strengthenging of the prime number theorem, is in a sense a probabilistic result. The Riemann hypothesis is, according to Tao, equivalent to the idea that the primes do behave randomly -- they are distributed according to the prime number theorem, with an error term that is exactly what you'd expect from the law of large numbers. This does not appear to be mentioned in, say, the Clay Mathematics Institute description of the problem, which I think is a shame.
All this got me thinking. I knew that there was a certain heuristic that is often used to determine what results about prime numbers "should be" true -- one assumes that a number which is approximately N has probability 1/log N of being prime. This works quite well, which led Paul Erdos to say: "God may not play dice with the universe, but something strange is going on with the prime numbers." This, for example, leads to a probabilistic argument for the Goldbach conjecture. The Goldbach conjecture states that every even integer greater than or equal to four is the sum of two primes: 4=2+2, 6=3+3, 8=3+5, 10=3+7=5+5, 12=5+7, 14=3+11=5+9, 16=3+13=5+11, 18=5+13=7+11, 20=3+17=7+13, and so on. For small numbers this seems like an accident. But there are, for example, six ways to write 100 as a sum of two primes:
100 = 3 + 97 = 11 + 89 = 17 + 83 = 29 + 71 = 41 + 59 = 47 + 53.
and if we pick a random good-sized even number there are lots of ways to write it as a sum of two primes. For example, the even numbers 4000, 4002, ..., 4020 can be written as a sum of two primes in 65, 106, 72, 52, 104, 65, 54, 108, 55, 66, 133 ways respectively. So we start to get the feeling that there are lots of ways to write even integers as sums of primes, and so it would be astonishingly shocking if the Great Mathematician In The Sky had stacked the deck so that, say, 4045580440 couldn't be witten as a sum of two primes.
On the other hand, there are infinitely many even integers. Let's say that by "astonishingly shocking" I meant that the "probability" that 2N can't be written as a sum of two primes is 1/N. Then, since the sum 1 + 1/2 + 1/3 + 1/4 + ... diverges, not only would Goldbach's conjecture be false, but it would be false infinitely many times.
As it turns out, though, we can make a guess at the probability that n can't be written as a sum of two primes. First, we try to estimate the number of ways to write n as the sum of two primes. This is the "probability" that 3 and n-3 are both prime, plus the "probability" that 4 and n-4 are both prime, plus the "probability" that 5 and n-5 are both prime, and so on up to the probability that n/2 and n/2 are both prime. (I know, you're saying "3 is prime! 4 isn't prime! What are you talking about?") Next, we assume that k and n-k being prime are independent, which isn't true. Then the expected number of ways that n can be written as a sum of two primes is
1/(log(3) log(n-3)) + 1/(log(4) log(n-4)) + ... + 1/(log(n/2) log(n/2))
which turns out to be about n/(2 log2 n). This is actually an underestimate, the actual value conjectured by Hardy and Littlewood being something like 1.3 n/(log2 n); the basic idea is that if k is prime, then n-k turns out to be more likely to be prime. For example, since n is even, and k must be odd, then n-k is odd, which makes it more likely to be prime. (This is pointed out at the Wikipedia article.) But even the independent result is quite strong. Since being prime is a "rare" property, I'll assume that the "distribution" (and I am using this word quite loosely) of the number of ways in which n can be written as a sum of two primes is a Poisson distribution with parameter n/(2 log2 n). This means that the "probability" that an even number n can't be written as a sum of two primes is
f(n) = exp(-n/(2 log2 n))
which is quite small for decent-sized numbers. For n = 10,000, for example, it's about 2.5 × 10-26. For n equal to one million, it's around 10-1138. The sum f(2) + f(4) + f(6) + ... converges to a number somewhere around 29, and what's more it converges quite quickly. So if there's a counterexample, it's small; if we can show that Goldbach is true for, say, numbers up to a million (which is easy to do with modern computers), then this heuristic shows it's "almost certainly" true.
There's a similar heuristic argument for why there are no odd perfect numbers, due to Pomerance; unfortunately this argument shows there are no even perfect numbers, as well. But it turns out that the even perfect numbers have a special sort of structure.
And one wants to know that the primes don't have this special sort of structure -- that they are in fact psuedorandom, as Tao calls it. (I'm not sure if "psuedorandom" is a well-defined term in any branch of mathematics; Tao uses it in two different ways in his talk, and explicitly points out that his two uses are different.)
Of course, this whole idea of thinking of primes probabilistically is sort of silly. In A Mathematician's Apology*, G. H. Hardy wrote that "317 is a prime, not because we think so, or because our minds are shaped in one way rather than another, but because it is so, because mathematical reality is built that way." Yet we don't know what the really big primes are. From a philosophical standpoint, why not act as if the primes are randomly distributed until we have evidence to the contrary? Proving the Riemann hypothesis would show that this way of looking at things is indeed valid.
A note on notation: when doing mathematics "log" means the natural logarithm; I can't think of a single place where the common, or base-10, logarithm appears in mathematics. Sometime base-2 logs appear, especially in questions in the analysis of algorithms where the analysis has some sort of structure that involves breaking things up in half. Base-10 logarithms are useful as a computational aid and nothing more. At one point I was teaching a calculus class and had a form on my web page where students could leave anonymous comments about the class. (I do not recommend this technique if you have a thin skin, but it seemed like a good idea at the time.) One of the students said, essentially, "please use 'ln' instead of 'log', it confuses me when you use 'log'. I didn't know how to respond to this. Fortunately, since the comment was anonymous, I didn't have to.)
Anyway, one of the ideas he mentions in this talk -- and this is something that I was not aware of before hearing this talk -- is that the Riemann hypothesis, which isequivalent to a certain strengthenging of the prime number theorem, is in a sense a probabilistic result. The Riemann hypothesis is, according to Tao, equivalent to the idea that the primes do behave randomly -- they are distributed according to the prime number theorem, with an error term that is exactly what you'd expect from the law of large numbers. This does not appear to be mentioned in, say, the Clay Mathematics Institute description of the problem, which I think is a shame.
All this got me thinking. I knew that there was a certain heuristic that is often used to determine what results about prime numbers "should be" true -- one assumes that a number which is approximately N has probability 1/log N of being prime. This works quite well, which led Paul Erdos to say: "God may not play dice with the universe, but something strange is going on with the prime numbers." This, for example, leads to a probabilistic argument for the Goldbach conjecture. The Goldbach conjecture states that every even integer greater than or equal to four is the sum of two primes: 4=2+2, 6=3+3, 8=3+5, 10=3+7=5+5, 12=5+7, 14=3+11=5+9, 16=3+13=5+11, 18=5+13=7+11, 20=3+17=7+13, and so on. For small numbers this seems like an accident. But there are, for example, six ways to write 100 as a sum of two primes:
100 = 3 + 97 = 11 + 89 = 17 + 83 = 29 + 71 = 41 + 59 = 47 + 53.
and if we pick a random good-sized even number there are lots of ways to write it as a sum of two primes. For example, the even numbers 4000, 4002, ..., 4020 can be written as a sum of two primes in 65, 106, 72, 52, 104, 65, 54, 108, 55, 66, 133 ways respectively. So we start to get the feeling that there are lots of ways to write even integers as sums of primes, and so it would be astonishingly shocking if the Great Mathematician In The Sky had stacked the deck so that, say, 4045580440 couldn't be witten as a sum of two primes.
On the other hand, there are infinitely many even integers. Let's say that by "astonishingly shocking" I meant that the "probability" that 2N can't be written as a sum of two primes is 1/N. Then, since the sum 1 + 1/2 + 1/3 + 1/4 + ... diverges, not only would Goldbach's conjecture be false, but it would be false infinitely many times.
As it turns out, though, we can make a guess at the probability that n can't be written as a sum of two primes. First, we try to estimate the number of ways to write n as the sum of two primes. This is the "probability" that 3 and n-3 are both prime, plus the "probability" that 4 and n-4 are both prime, plus the "probability" that 5 and n-5 are both prime, and so on up to the probability that n/2 and n/2 are both prime. (I know, you're saying "3 is prime! 4 isn't prime! What are you talking about?") Next, we assume that k and n-k being prime are independent, which isn't true. Then the expected number of ways that n can be written as a sum of two primes is
1/(log(3) log(n-3)) + 1/(log(4) log(n-4)) + ... + 1/(log(n/2) log(n/2))
which turns out to be about n/(2 log2 n). This is actually an underestimate, the actual value conjectured by Hardy and Littlewood being something like 1.3 n/(log2 n); the basic idea is that if k is prime, then n-k turns out to be more likely to be prime. For example, since n is even, and k must be odd, then n-k is odd, which makes it more likely to be prime. (This is pointed out at the Wikipedia article.) But even the independent result is quite strong. Since being prime is a "rare" property, I'll assume that the "distribution" (and I am using this word quite loosely) of the number of ways in which n can be written as a sum of two primes is a Poisson distribution with parameter n/(2 log2 n). This means that the "probability" that an even number n can't be written as a sum of two primes is
f(n) = exp(-n/(2 log2 n))
which is quite small for decent-sized numbers. For n = 10,000, for example, it's about 2.5 × 10-26. For n equal to one million, it's around 10-1138. The sum f(2) + f(4) + f(6) + ... converges to a number somewhere around 29, and what's more it converges quite quickly. So if there's a counterexample, it's small; if we can show that Goldbach is true for, say, numbers up to a million (which is easy to do with modern computers), then this heuristic shows it's "almost certainly" true.
There's a similar heuristic argument for why there are no odd perfect numbers, due to Pomerance; unfortunately this argument shows there are no even perfect numbers, as well. But it turns out that the even perfect numbers have a special sort of structure.
And one wants to know that the primes don't have this special sort of structure -- that they are in fact psuedorandom, as Tao calls it. (I'm not sure if "psuedorandom" is a well-defined term in any branch of mathematics; Tao uses it in two different ways in his talk, and explicitly points out that his two uses are different.)
Of course, this whole idea of thinking of primes probabilistically is sort of silly. In A Mathematician's Apology*, G. H. Hardy wrote that "317 is a prime, not because we think so, or because our minds are shaped in one way rather than another, but because it is so, because mathematical reality is built that way." Yet we don't know what the really big primes are. From a philosophical standpoint, why not act as if the primes are randomly distributed until we have evidence to the contrary? Proving the Riemann hypothesis would show that this way of looking at things is indeed valid.
A note on notation: when doing mathematics "log" means the natural logarithm; I can't think of a single place where the common, or base-10, logarithm appears in mathematics. Sometime base-2 logs appear, especially in questions in the analysis of algorithms where the analysis has some sort of structure that involves breaking things up in half. Base-10 logarithms are useful as a computational aid and nothing more. At one point I was teaching a calculus class and had a form on my web page where students could leave anonymous comments about the class. (I do not recommend this technique if you have a thin skin, but it seemed like a good idea at the time.) One of the students said, essentially, "please use 'ln' instead of 'log', it confuses me when you use 'log'. I didn't know how to respond to this. Fortunately, since the comment was anonymous, I didn't have to.)
14 July 2007
If you're going to lie with numbers, at least be subtle about it
Mark C. Chu-Carroll at Good Math, Bad Math comments on A Laughable Laffer Curve from the WSJ.
In economics there is the Laffer curve, which posits that increasing tax rates increases government revenue up to a certain tax rate, after which increasing taxes actually decreases revenue because it discourages people from producing wealth. A lot of macroeconomics is devoted to determining what the Laffer curve looks like, in order to argue that taxes should be raised/lowered. Chu-Carroll explains it better than I can; you should read his post.
The Wall Street Journal is far to the right economically, and shows this by drawing a "curve" fit to a bunch of points that is clearly contrived to make it look like lowering corporate taxes in the US is a Good Idea. I know that I don't know enough economics to comment. But I do know that the WSJ could have been a little more subtle.
In economics there is the Laffer curve, which posits that increasing tax rates increases government revenue up to a certain tax rate, after which increasing taxes actually decreases revenue because it discourages people from producing wealth. A lot of macroeconomics is devoted to determining what the Laffer curve looks like, in order to argue that taxes should be raised/lowered. Chu-Carroll explains it better than I can; you should read his post.
The Wall Street Journal is far to the right economically, and shows this by drawing a "curve" fit to a bunch of points that is clearly contrived to make it look like lowering corporate taxes in the US is a Good Idea. I know that I don't know enough economics to comment. But I do know that the WSJ could have been a little more subtle.
bracketings and triple negatives
From Language Log: some commentary on yesterday's Doonesbury. The text is as follows:
Some guy: I just don't get it, Jorge. Why do you get the big bucks and not me?
Jorge: We have different work styles, man.
Some guy: Like how?
Jorge: Well, for one thing, I'm not stoned half the time.
Some guy: What are you saying -- that I am?
Jorge: I'm not saying that you're not.
Some guy: Don't try your tricky double negatives on me, señor!
Jorge: Enjoy your break.
Some guy: What?
Jorge: Never mind.
Multiple negatives are indeed tricky. As the post at Language Log points out, a lot of the time a double negative is indeed a negative, and triple negatives often turn out to be positives. The triple negatives they give examples of are generally obtained by taking a double negative which has negative semantics and then negating it again.
And our main character in this particular strip, whose name I don't know, could argue that even if he is stoned half the time, he's also not-stoned half the time. The following conversation could take place:
Jorge: Well, for one thing, [I'm not] [stoned half the time].
Some guy: I'm [not stoned] half the time.
where the same string of words occurs, but with two different meanings. In general there is quite a large number of ways to put parentheses around the words in a string like this, which is a semi-standard tactic for linguistic analysis. (It seems to me that linguists, when talking about syntax, like to draw trees when they want to explain how a whole sentence works, but they do ad-hoc bracketings like the one I did above when they just want to make a simple point about which words go with which words, because trees are annoying to draw.) The number of ways to bracket an n-word sentence completely turns out to be the (n-1)st Catalan number; a lot of problems involving trees (and various other recursive structures!) end up involving Catalan numbers. Of course, most of these bracketings are illogical: anything that looked like
In general, I suspect mathematicians tend to parse natural-language sentences differently than "ordinary people", because we are more used to dealing with subjects where the precise meaning of some statement is what matters, and an interpretation which is off by a little bit might as well be no interpretation at all. (Or perhaps worse than no interpretation -- it is better to not understand a complicated definition and know you don't understand it than to not understand it but think you do. The truly wise are aware of the limits of their knowledge.) The Jargon File, in referring to the speech style of the hacker community (which has some overlap and contact with the mathematical community), states that "...English-speaking hackers almost never use double negatives, even if they live in a region where colloquial usage allows them. The thought of uttering something that logically ought to be an affirmative knowing it will be misparsed as a negative tends to disturb them."
Some guy: I just don't get it, Jorge. Why do you get the big bucks and not me?
Jorge: We have different work styles, man.
Some guy: Like how?
Jorge: Well, for one thing, I'm not stoned half the time.
Some guy: What are you saying -- that I am?
Jorge: I'm not saying that you're not.
Some guy: Don't try your tricky double negatives on me, señor!
Jorge: Enjoy your break.
Some guy: What?
Jorge: Never mind.
Multiple negatives are indeed tricky. As the post at Language Log points out, a lot of the time a double negative is indeed a negative, and triple negatives often turn out to be positives. The triple negatives they give examples of are generally obtained by taking a double negative which has negative semantics and then negating it again.
And our main character in this particular strip, whose name I don't know, could argue that even if he is stoned half the time, he's also not-stoned half the time. The following conversation could take place:
Jorge: Well, for one thing, [I'm not] [stoned half the time].
Some guy: I'm [not stoned] half the time.
where the same string of words occurs, but with two different meanings. In general there is quite a large number of ways to put parentheses around the words in a string like this, which is a semi-standard tactic for linguistic analysis. (It seems to me that linguists, when talking about syntax, like to draw trees when they want to explain how a whole sentence works, but they do ad-hoc bracketings like the one I did above when they just want to make a simple point about which words go with which words, because trees are annoying to draw.) The number of ways to bracket an n-word sentence completely turns out to be the (n-1)st Catalan number; a lot of problems involving trees (and various other recursive structures!) end up involving Catalan numbers. Of course, most of these bracketings are illogical: anything that looked like
I'm not [stoned half] the timejust isn't going to correspond to a grammatical interpretation of that sentence, because "stoned half" doesn't mean anything. But I suspect that longer sentences have more possible interpretations, on average; it's kind of surprising that a six-word sentence can actually be parsed in multiple ways.
In general, I suspect mathematicians tend to parse natural-language sentences differently than "ordinary people", because we are more used to dealing with subjects where the precise meaning of some statement is what matters, and an interpretation which is off by a little bit might as well be no interpretation at all. (Or perhaps worse than no interpretation -- it is better to not understand a complicated definition and know you don't understand it than to not understand it but think you do. The truly wise are aware of the limits of their knowledge.) The Jargon File, in referring to the speech style of the hacker community (which has some overlap and contact with the mathematical community), states that "...English-speaking hackers almost never use double negatives, even if they live in a region where colloquial usage allows them. The thought of uttering something that logically ought to be an affirmative knowing it will be misparsed as a negative tends to disturb them."
the Golden Ratio sells pants?
From mathtrek and Gooseania I have learned that a company called The ProportionofBlu is making jeans which they claim are designed based on the golden ratio.
They say, on their (annoyingly flashful) web site:
The claims about the Golden Ratio being ubiquitous have been debunked (read the Mathtrek post for a start), so it's kind of funny to see the fashion people using it.
The Golden Ratio, by the way, is the positive solution to the equation x2 = x + 1, which is (1 + √5)/2, or about 1.618. It has the property that it's equal to 1 plus its own reciprocal, which I'll use later.
But there are probably some ridiculously large number of measurements that go into making a pair of jeans. It would surprise me more if they managed to avoid having any of these measurements be in a 1.618:1 ratio to each other. This is especially true because jeans come in different sizes, and the measurements that are in this ratio in a size 4 won't be in this ratio in a size 14. (Then again, these people may not even make size 14 jeans, rendering the whole issue moot.) I have a strong suspicion that the golden ratio was not used at all in the design of these jeans, but they figured that by slapping this psuedo-mystical copy into the ads they could raise their prices. And you know what? They're probably right. Lots of smart people work in advertising.
Then again, the golden ratio arises naturally if you look at a five-pointed star, or a regular pentagon. In the diagram at left, taken from the Wikipedia article "Pentagram",
, consider the four lengths indicated by the red, green, blue, and magenta line segments; the ratio between each adjacent pair of these is the golden ratio. The golden ratio also arises in construction of the dodecahedron (which has twelve pentagons as faces) and the icosahedron (which is "dual" to the dodecahedron). Since pentagons have golden ratios embedded in them in this way, you could argue that the United States Department of Defense is built around the golden ratio.
They say, on their (annoyingly flashful) web site:
At the core of every ProportionofBlu creation lies the "Golden Ratio"; a naturally reoccurring sequence of measurements and patterns found in the very building blocks of all life. From the most intricate designs in nature to the most sublime achievements of man, the ratio manifests itself. It lies within the air vortex of a bird's wing, the sensuous curves of a Stradivarius Violin, the structural design of the Great Pyramids and the double helix that houses our very DNA.
It's about form, fit, and functionality. It's about the beauty of everybody and everything. Meticulously hand-crafted, every ProportionofBlu creation is designed "in the ratio" just for you.
The claims about the Golden Ratio being ubiquitous have been debunked (read the Mathtrek post for a start), so it's kind of funny to see the fashion people using it.
The Golden Ratio, by the way, is the positive solution to the equation x2 = x + 1, which is (1 + √5)/2, or about 1.618. It has the property that it's equal to 1 plus its own reciprocal, which I'll use later.
But there are probably some ridiculously large number of measurements that go into making a pair of jeans. It would surprise me more if they managed to avoid having any of these measurements be in a 1.618:1 ratio to each other. This is especially true because jeans come in different sizes, and the measurements that are in this ratio in a size 4 won't be in this ratio in a size 14. (Then again, these people may not even make size 14 jeans, rendering the whole issue moot.) I have a strong suspicion that the golden ratio was not used at all in the design of these jeans, but they figured that by slapping this psuedo-mystical copy into the ads they could raise their prices. And you know what? They're probably right. Lots of smart people work in advertising.
Then again, the golden ratio arises naturally if you look at a five-pointed star, or a regular pentagon. In the diagram at left, taken from the Wikipedia article "Pentagram",
, consider the four lengths indicated by the red, green, blue, and magenta line segments; the ratio between each adjacent pair of these is the golden ratio. The golden ratio also arises in construction of the dodecahedron (which has twelve pentagons as faces) and the icosahedron (which is "dual" to the dodecahedron). Since pentagons have golden ratios embedded in them in this way, you could argue that the United States Department of Defense is built around the golden ratio.
13 July 2007
million-dollar waitress update
A few weeks ago I wrote about Mary Sue Williams, who at the time looked like the probable winner of a CNBC stock-picking contest. Basically, she was the best out of those people who hadn't "cheated".
MSNBC reports that she has been declared the winner and presented with an oversized novelty check.
MSNBC reports that she has been declared the winner and presented with an oversized novelty check.
math is not a magic bullet
From Slashdot: Optimum Copyright Period Decided By Math.
The idea is interesting -- copyright periods should be chosen to maximize some quantity (I think it's called "welfare" in the original paper), and this paper does it. It's basically a theoretical economics paper, and it looks accessible to someone with first-year calculus and common sense under their belt. I won't criticize the paper, because it's twenty-nine pages and I haven't read it.
What bothers me is the headline. Mathematics is not some sort of magic bullet! The paper makes a lot of assumptions about things in the "real world" which may or may not be true. Just because this thing spits out "14 years" doesn't mean that's the optimal copyright period -- see page 24 of the paper, which states
Those are a lot of arbitrary-sounding constants. If they were a little different, would the answer be different? Of course.
It does, however, suggest that the current copyright periods of, say, life of the author plus fifty years are too long.
The idea is interesting -- copyright periods should be chosen to maximize some quantity (I think it's called "welfare" in the original paper), and this paper does it. It's basically a theoretical economics paper, and it looks accessible to someone with first-year calculus and common sense under their belt. I won't criticize the paper, because it's twenty-nine pages and I haven't read it.
What bothers me is the headline. Mathematics is not some sort of magic bullet! The paper makes a lot of assumptions about things in the "real world" which may or may not be true. Just because this thing spits out "14 years" doesn't mean that's the optimal copyright period -- see page 24 of the paper, which states
With α = 0.12, γ = 0.129, then θ ≈ 0.93. With our defaults of a discount rate of 6% and cultural decay of 5% this implies an optimal copyright term of just over 14 years.
Those are a lot of arbitrary-sounding constants. If they were a little different, would the answer be different? Of course.
It does, however, suggest that the current copyright periods of, say, life of the author plus fifty years are too long.
roller-coaster of a week: Friday the 13th
Just six days after "the luckiest day of the century", we have Friday the 13th, supposedly the unluckiest day.
The 13th day of a month occurs on Friday more often than on any other day. But it's not that often -- 688 times every 400 years. You'd expect 4800/7 = 685.714... times.
Perhaps there should be a tradition of, say, Wednesday the 25th being a day of good luck. (Let's see -- maybe Saint Nicholas was born on a Wednesday, and the 25th of December is Christmas...) Of course Friday the 13ths and Wednesday the 25ths would always occur in the same month. I wonder how long it would take the average person to realize that the good-luck-day and the bad-luck-day always happening in the same month was more than a coincidence. The most likely day for Christmas, by the way, is not Wednesday -- it's a tie between Sunday, Tuesday, and Friday.
I'd heard that the 13th occurred most commonly on a Friday before. A few weeks ago I tried to compute the probabilities in my head. Unfortunately they are all so close to one-seventh that I couldn't, at least not while walking.
To calculate days of the week in general, in your head, you can use the Doomsday Algorithm, created by J. H. Conway. I seem to recall reading once that Conway had his computer programmed so that he couldn't log onto it unless he could determine the day of the week corresponding to a given date within some (short) amount of time; this may or may not be true.
Last but not least, the Phillies. I have tickets tonight. They will lose, although I don't want them to, and that will be their 10,000th loss. How do I know this? Because they're playing the Cardinals. And game four of the 2004 World Series -- the one the Red Sox won in four game -- was played against the Cardinals, during a total lunar eclipse. Historic things happen when the Cardinals are playing and there's already some sort of bad omen going on.
(Note to the stupid: I don't really believe the previous paragraph.)
The 13th day of a month occurs on Friday more often than on any other day. But it's not that often -- 688 times every 400 years. You'd expect 4800/7 = 685.714... times.
Perhaps there should be a tradition of, say, Wednesday the 25th being a day of good luck. (Let's see -- maybe Saint Nicholas was born on a Wednesday, and the 25th of December is Christmas...) Of course Friday the 13ths and Wednesday the 25ths would always occur in the same month. I wonder how long it would take the average person to realize that the good-luck-day and the bad-luck-day always happening in the same month was more than a coincidence. The most likely day for Christmas, by the way, is not Wednesday -- it's a tie between Sunday, Tuesday, and Friday.
I'd heard that the 13th occurred most commonly on a Friday before. A few weeks ago I tried to compute the probabilities in my head. Unfortunately they are all so close to one-seventh that I couldn't, at least not while walking.
To calculate days of the week in general, in your head, you can use the Doomsday Algorithm, created by J. H. Conway. I seem to recall reading once that Conway had his computer programmed so that he couldn't log onto it unless he could determine the day of the week corresponding to a given date within some (short) amount of time; this may or may not be true.
Last but not least, the Phillies. I have tickets tonight. They will lose, although I don't want them to, and that will be their 10,000th loss. How do I know this? Because they're playing the Cardinals. And game four of the 2004 World Series -- the one the Red Sox won in four game -- was played against the Cardinals, during a total lunar eclipse. Historic things happen when the Cardinals are playing and there's already some sort of bad omen going on.
(Note to the stupid: I don't really believe the previous paragraph.)
ten thousand pennies
Philadelphia Fish & Company is running a promotion in which, once the Phillies lose their ten-thousandth game, anybody who brings in ten thousand pennies can get a dinner for ten.
This is from the Don Polec's World segment on WPVI's 6 PM newscast yesterday. (To see the actual segment, look at the menu under the "Action News on demand".) The guy who came up with the promotion, Kevin Meeker, says it comes from an ancient Greek tradition -- when the ancient Greeks had lost ten thousand men in battle, they had a feast of fish and it was believed to bring them good luck.
(To answer the obvious question -- you actually have to bring ten thousand pennies. You can't just show up with a $100 bill. And they have to be rolled. He wants people to suffer.)
The Phillies are currently at 9999 losses. The probability of the Phillies' ten-thousandth loss tonight, which I originally looked at here and re-examined here and here? Just under 40%. The full distribution -- which is now nothing more than the distribution of the time until the team's next loss is as follows:
And how much do ten thousand pennies weigh? Twenty-five kilograms, or about fifty-five pounds.
This is from the Don Polec's World segment on WPVI's 6 PM newscast yesterday. (To see the actual segment, look at the menu under the "Action News on demand".) The guy who came up with the promotion, Kevin Meeker, says it comes from an ancient Greek tradition -- when the ancient Greeks had lost ten thousand men in battle, they had a feast of fish and it was believed to bring them good luck.
(To answer the obvious question -- you actually have to bring ten thousand pennies. You can't just show up with a $100 bill. And they have to be rolled. He wants people to suffer.)
The Phillies are currently at 9999 losses. The probability of the Phillies' ten-thousandth loss tonight, which I originally looked at here and re-examined here and here? Just under 40%. The full distribution -- which is now nothing more than the distribution of the time until the team's next loss is as follows:
| Jul 13 | v. Cardinals | 0.394329 |
| Jul 14 | v. Cardinals | 0.238834 |
| Jul 15 | v. Cardinals | 0.144655 |
| Jul 16 | @ Dodgers | 0.130046 |
| Jul 17 | @ Dodgers | 0.053929 |
| Jul 18 | @ Dodgers | 0.022364 |
| Jul 19 | @ Padres | 0.009184 |
| Jul 20 | @ Padres | 0.003861 |
| Jul 21 | @ Padres | 0.001623 |
| Jul 22 | @ Padres | 0.000682 |
| Jul 24 | v. Nationals | 0.000174 |
| Jul 25 | v. Nationals | 0.000113 |
| Jul 26 | v. Nationals | 0.000073 |
| Jul 27 | v. Pirates | 0.000048 |
| Jul 28 | v. Pirates | 0.000031 |
| Jul 29 | v. Pirates | 0.000020 |
| Jul 30 | @ Cubs | 0.000018 |
| Jul 31 | @ Cubs | 0.000009 |
| Aug 01 | @ Cubs | 0.000004 |
| Aug 02 | @ Cubs | 0.000002 |
| Aug 03 | @ Brewers | 0.000001 |
| Aug 04 | @ Brewers | 0.000001 |
12 July 2007
why Four Corners is in the middle of nowhere
Mark Chu at ScienceBlogs writes about fractals. The classic example of a fractal is the idea that it is difficult to measure the length of the border between two countries -- Chu's example is Portugal and Spain -- because the length of the border depends on the scale at which you measure it.
In technical terms, the border is nonrectifiable. For those of you who don't know that word, basically what you should know is that the only thing we can "really" find the length of is a straight line segment. To determine the length of any curve* which isn't a straight line segments, we approximate it by straight line segments and add up their length to get an approximation of the legnth of our original curve. For a circle, this means that we approximate the circle as a square, an octagon, a 16-gon, a 32-gon and so on. The square has length the ancient Greek mathematician Archimedes knew how to do this, and he famously showed that π, the circumference (i. e. length) of a circle of diameter 1 was between 3+10/71 and 3+1/7. To four decimal places, these are 3.1408 and 3.1429, respectively; the actual value is about 3.1416.
The total lengths of the square, octagon, 16-gon, and 32-gon are 2.8284, 3.0615, 3.1214, 3.1365 -- you can see how these approach 3.1416.
But for a lot of the objects that exist in "real life", that sequence doesn't approach anything. You might see that each time you halve the length of the line -- which corresponds to doubling the number of sides in the example above -- the length increases by 10%. As you measure on finer and finer scales, the length gets longer and longer.
Natural features tend to have this sort of behavior -- mountains are not cones, clouds are not spheres, and so on -- while artifical features tend to be straight lines or curves with simple mathematical definitions. Take a look at a map of the United States, for example. The eastern states for the most part have boundaries which are natural features -- rivers, the crests of mountain ranges, and so on -- because rivers and mountains were the natural barriers that separated population centers. The western states, which were divided up for the most part before they were settled and by people who didn't have much of an idea of the population patterns anyway, are often straight lines. There's a reason that "Four Corners", the point where Utah, Colorado, Arizona, and New Mexico, is remote. (Link goes to a Google map/satellite image.)
* The mathematical use of "curve" includes what normal people would call "curves", but also lots of other things. For example, a straight line is a "curve" to a mathematician. It is an exceedingly boring curve, but it is still a curve.
In technical terms, the border is nonrectifiable. For those of you who don't know that word, basically what you should know is that the only thing we can "really" find the length of is a straight line segment. To determine the length of any curve* which isn't a straight line segments, we approximate it by straight line segments and add up their length to get an approximation of the legnth of our original curve. For a circle, this means that we approximate the circle as a square, an octagon, a 16-gon, a 32-gon and so on. The square has length the ancient Greek mathematician Archimedes knew how to do this, and he famously showed that π, the circumference (i. e. length) of a circle of diameter 1 was between 3+10/71 and 3+1/7. To four decimal places, these are 3.1408 and 3.1429, respectively; the actual value is about 3.1416.
The total lengths of the square, octagon, 16-gon, and 32-gon are 2.8284, 3.0615, 3.1214, 3.1365 -- you can see how these approach 3.1416.
But for a lot of the objects that exist in "real life", that sequence doesn't approach anything. You might see that each time you halve the length of the line -- which corresponds to doubling the number of sides in the example above -- the length increases by 10%. As you measure on finer and finer scales, the length gets longer and longer.
Natural features tend to have this sort of behavior -- mountains are not cones, clouds are not spheres, and so on -- while artifical features tend to be straight lines or curves with simple mathematical definitions. Take a look at a map of the United States, for example. The eastern states for the most part have boundaries which are natural features -- rivers, the crests of mountain ranges, and so on -- because rivers and mountains were the natural barriers that separated population centers. The western states, which were divided up for the most part before they were settled and by people who didn't have much of an idea of the population patterns anyway, are often straight lines. There's a reason that "Four Corners", the point where Utah, Colorado, Arizona, and New Mexico, is remote. (Link goes to a Google map/satellite image.)
* The mathematical use of "curve" includes what normal people would call "curves", but also lots of other things. For example, a straight line is a "curve" to a mathematician. It is an exceedingly boring curve, but it is still a curve.
publishers using trumped-up focus groups
The Science of Success, by James Surowiecki, author of Wisdom of Crowds
(via Freakonomics).
Apparently Simon and Schuster has partnered with a company called MediaPredict to set up a system in which consumers can evaluate book proposals.
This seems to me like one of the least surprising uses of prediction markets, because S&S is doing the evaluation and their purpose here is not to determine the book that the critics will like most, but the one that the people will like most. So why not ask the people? It strikes me as a fancy version of a focus group.
(By the way, "The Wisdom of Crowds" is difficult to find on Amazon if you don't limit the search for books, because Amazon assumes that "crowds" is a misspelling of "cords" and finds me all sorts of cordless devices. Sure, Surowiecki's book is a cordless device, but aren't all books?)
(via Freakonomics).Apparently Simon and Schuster has partnered with a company called MediaPredict to set up a system in which consumers can evaluate book proposals.
This seems to me like one of the least surprising uses of prediction markets, because S&S is doing the evaluation and their purpose here is not to determine the book that the critics will like most, but the one that the people will like most. So why not ask the people? It strikes me as a fancy version of a focus group.
(By the way, "The Wisdom of Crowds" is difficult to find on Amazon if you don't limit the search for books, because Amazon assumes that "crowds" is a misspelling of "cords" and finds me all sorts of cordless devices. Sure, Surowiecki's book is a cordless device, but aren't all books?)
can scientists make up their minds?
How should unproven findings be publicized? at Statistical Modeling, Causal Inference, and Social Science, via Notional Slurry.
A year or so ago it was claimed by Satoshi Kanazawa that, roughly, attractive people are more likely to have daughters than sons. I've also heard recently that successful people are more likely to have sons; for example, something like sixty percent of all the children of U.S. presidents have been male. The mass media have recently picked up this story. Andrew at Statistical Modeling, Causal Inference, and Social Science criticized this finding; you can read his commentary to find out why, but he believes that the findings are statistical artifacts. However, he thinks there may be some truth to the conjectures
Of course, any given mass media outlet isn't going to report "maybe attractive people have more daughters than sons". They'll report one of the two following things:
Then in the first case they'd go and find an attractive couple that had, say, three sons and no daughters, to "disprove" this.
This reminds me of the way that the mass media treats, say, dark chocolate. Chocolate's supposed to be bad for you, because it is full of fat and sugar. But it's also supposed to be good for you because it contains certain antioxidants. One day they'll report one thing, one day they'll report the other. You know what I do? I ignore all the studies and eat chocolate, because I like it. (I happen to prefer dark chocolate to milk chocolate, which is probably a good thing in terms of health, but my preference is motivated purely by taste.)
This probably leads laypeople to have the idea that scientists are constantly changing their minds (which is true -- good scientists change their minds as new evidence comes in, or as new interpretations for old evidence become clear). I fear, though, that this may also lead to laypeople distrusting science -- if they can't even decide whether chocolate is "good for you" or "bad for you", what good are they?
But science is more complicated than that -- no food is entirely "good" or "bad", and so on.
What I'd like to see -- and perhaps it's already out there -- is more reporting of the meta-literature. Not "one group of scientists said today that chocolate is good", or "another group of scientists said today that chocolate is bad", but "some scientist looked at what all the other scientists said, and most of them think chocolate is bad". But you're not going to hear that on the evening news because the chocolate maker advertises there. But what about aggregating all that stuff online?
But I think most people will want a one-bit answer -- "yes" or "no". And it's more complicated than that. Some studies don't even give you a whole bit of information -- they tell you "probably yes". And a bunch of these "probably yes" answers can add up to a "yes". But not if everybody's working in isolation. This applies to ordinary life as well -- and I believe that it would be useful if there were some way that all the anecdotal experiences of people with, say, a particular company could be aggregated into something statistically significant. But that's a matter for another post.
A year or so ago it was claimed by Satoshi Kanazawa that, roughly, attractive people are more likely to have daughters than sons. I've also heard recently that successful people are more likely to have sons; for example, something like sixty percent of all the children of U.S. presidents have been male. The mass media have recently picked up this story. Andrew at Statistical Modeling, Causal Inference, and Social Science criticized this finding; you can read his commentary to find out why, but he believes that the findings are statistical artifacts. However, he thinks there may be some truth to the conjectures
Of course, any given mass media outlet isn't going to report "maybe attractive people have more daughters than sons". They'll report one of the two following things:
- Scientists say that attractive people have more daughters than sons.
- Scientists say that there is no link between attractiveness and how many daughters you have.
Then in the first case they'd go and find an attractive couple that had, say, three sons and no daughters, to "disprove" this.
This reminds me of the way that the mass media treats, say, dark chocolate. Chocolate's supposed to be bad for you, because it is full of fat and sugar. But it's also supposed to be good for you because it contains certain antioxidants. One day they'll report one thing, one day they'll report the other. You know what I do? I ignore all the studies and eat chocolate, because I like it. (I happen to prefer dark chocolate to milk chocolate, which is probably a good thing in terms of health, but my preference is motivated purely by taste.)
This probably leads laypeople to have the idea that scientists are constantly changing their minds (which is true -- good scientists change their minds as new evidence comes in, or as new interpretations for old evidence become clear). I fear, though, that this may also lead to laypeople distrusting science -- if they can't even decide whether chocolate is "good for you" or "bad for you", what good are they?
But science is more complicated than that -- no food is entirely "good" or "bad", and so on.
What I'd like to see -- and perhaps it's already out there -- is more reporting of the meta-literature. Not "one group of scientists said today that chocolate is good", or "another group of scientists said today that chocolate is bad", but "some scientist looked at what all the other scientists said, and most of them think chocolate is bad". But you're not going to hear that on the evening news because the chocolate maker advertises there. But what about aggregating all that stuff online?
But I think most people will want a one-bit answer -- "yes" or "no". And it's more complicated than that. Some studies don't even give you a whole bit of information -- they tell you "probably yes". And a bunch of these "probably yes" answers can add up to a "yes". But not if everybody's working in isolation. This applies to ordinary life as well -- and I believe that it would be useful if there were some way that all the anecdotal experiences of people with, say, a particular company could be aggregated into something statistically significant. But that's a matter for another post.
Euler: The Master of Us All
In honor of Leonhard Euler's 300th birthday: the Sudoku watch, in a limited 1707-piece edition. (Euler was born in 1707, hence the number.) For some reason it costs $1678, not $1707. (And $2268 is not 1707 euros; I checked.) Euler didn't invent Sudoku, but he did work on Latin squares, of which Sudoku are a special case. (Thanks to Amy for pointing me to this.)
In other Eulerian news (how often do you get to use that phrase?), I just finished reading Euler: The Master of Us All
by Wiliam Dunham, who is interested in the history of mathematics. I'd read his previous book Journey through Genius: The Great Theorems of Mathematics
when I was in high school; this book treats a dozen or so of the major theorems of mathematics, one to a chapter, in a way that's probably accessible to the bright high school student. His book on Euler is a bit trickier -- it assumes one is familiar with the calculus and comfortable with infinite series. Euler was very comfortable with infinite series; this is what enabled, for example, his solution of the Basel problem: finding the sum 1 + 1/4 + 1/9 + 1/16 + ..., which turns out to be π2/6. The Wikipedia article gives two proofs, Euler's proof and a more rigorous one. Personally, I prefer Euler's proof; I often find that the rigor of modern proofs obscures why the result in question is actually true.
Dunham's book treats some of Euler's major results:
1 + 1/2 + 1/3 + 1/4 + 1/5 + ... = (2 x 3 x 5 x 7 x 11 x 13 x ...)/(1 x 2 x 4 x 6 x 10 x 12 x ...)
which seems silly, because both sides are infinite. But it makes sense. We have, for example,
1 + 1/2 + 1/4 + 1/8 + 1/16 + ... = 2/1
and then
1 + 1/2 + 1/3 + 1/4 + 1/6 + 1/8 + 1/9 + 1/12 + ...
= (1 + 1/2 + 1/4 + 1/8 + ...) (1 + 1/3 + 1/9 + 1/27 + ...)
= (2/1) (3/2)
where the original sum is the sum of reciprocals of numbers whose only prime factors are 2 and 3. Why not extend this to the sum of the reciprocals of all integers using unique factorization? And it turns out that this sum has a perfectly reasonable interpretation: by the same sort of argument as above,
1 + 1/2s + 1/3s + 1/4s + ... = 1/[(1-2-s)(1-3-s)(1-5-s)] ...
where if s>1 both sides converge; if you let s approach 1 from above you recover Euler's result. (This is due to Kronecker, and can be found on pp. 66-70 of Dunham.) But this sort of heuristic appeals to me. It seems to me that modern mathematicians don't take enough risks like this. Or, if they do, they are not telling me; I fear that the papers currently being published omit anything which isn't totally rigorous, even if (especially if?) it might make things clearer. I suppose this makes sense if you want to keep your papers short, which was of course important when papers were printed on, well, paper -- but these sorts of less rigorous things could certainly be made available in some sort of supplement to one's paper, published on a web page, for example.
Finally, given how many areas Euler has touched, sometimes I've joked that we should just rename mathematics "Euler". Somewhat more seriously, though -- I wonder to what extent one can tell whether someone is a mathematician by writing the word "Euler" on a piece of paper and asking them to pronounce it. (Isaac Asimov, who was trained as a chemist, suggested that "unionized" played the same role for chemists. Chemists pronounce it in four syllables and think it refers to molecules which have the same number of protons as electrons and thus a neutral charge; non-chemists pronounce it in three syllables and think it refers to organized labor.)
(My apologies for the awkwardness of the notation here. There seem to be ways to write out mathematics in blog posts -- either using LaTeX or MathML -- but they're not necessary for the notation I want to use here and seem like they'd take a little fooling around to get to work. And I don't even know MathML, although I wouldn't mind learning it.)
In other Eulerian news (how often do you get to use that phrase?), I just finished reading Euler: The Master of Us All
by Wiliam Dunham, who is interested in the history of mathematics. I'd read his previous book Journey through Genius: The Great Theorems of Mathematics when I was in high school; this book treats a dozen or so of the major theorems of mathematics, one to a chapter, in a way that's probably accessible to the bright high school student. His book on Euler is a bit trickier -- it assumes one is familiar with the calculus and comfortable with infinite series. Euler was very comfortable with infinite series; this is what enabled, for example, his solution of the Basel problem: finding the sum 1 + 1/4 + 1/9 + 1/16 + ..., which turns out to be π2/6. The Wikipedia article gives two proofs, Euler's proof and a more rigorous one. Personally, I prefer Euler's proof; I often find that the rigor of modern proofs obscures why the result in question is actually true.Dunham's book treats some of Euler's major results:
- the Euclid-Euler theorem on perfect numbers: all even perfect numbers have the form (2k-1)2k-1, where 2k-1 is prime. Odd perfect numbers are harder to find; Carl Pomerance has a heuristic argument that they don't exist.
- infinite series involving logarithms; apparently Euler was the first to look at logarithms as the inverse of the exponential function, as opposed to just a useful computational aid.
- the solution to the Basel problem and some related series; in fact Dunham talked about this in Journey through Genius as well, which is where I first learned about it (edit, 7/13/07: you can read more about the Basel problem, which is the summing of the series 1 + 1/4 + 1/9 + 1/16 + ... = π2/6, at the Everything Seminar
- the solution of third- and fourth-degree algebraic equations. (Did you know that any quartic polynomial with real coefficients can be factored as the product of two quadratics with real coefficients? I didn't.)
- some old-fashioned synthetic geometry: Heron's formula for the area of a triangle, for example
- solutions of the derangement problem and the fact that partitions into odd parts and into distinct parts are equinumerous
1 + 1/2 + 1/3 + 1/4 + 1/5 + ... = (2 x 3 x 5 x 7 x 11 x 13 x ...)/(1 x 2 x 4 x 6 x 10 x 12 x ...)
which seems silly, because both sides are infinite. But it makes sense. We have, for example,
1 + 1/2 + 1/4 + 1/8 + 1/16 + ... = 2/1
and then
1 + 1/2 + 1/3 + 1/4 + 1/6 + 1/8 + 1/9 + 1/12 + ...
= (1 + 1/2 + 1/4 + 1/8 + ...) (1 + 1/3 + 1/9 + 1/27 + ...)
= (2/1) (3/2)
where the original sum is the sum of reciprocals of numbers whose only prime factors are 2 and 3. Why not extend this to the sum of the reciprocals of all integers using unique factorization? And it turns out that this sum has a perfectly reasonable interpretation: by the same sort of argument as above,
1 + 1/2s + 1/3s + 1/4s + ... = 1/[(1-2-s)(1-3-s)(1-5-s)] ...
where if s>1 both sides converge; if you let s approach 1 from above you recover Euler's result. (This is due to Kronecker, and can be found on pp. 66-70 of Dunham.) But this sort of heuristic appeals to me. It seems to me that modern mathematicians don't take enough risks like this. Or, if they do, they are not telling me; I fear that the papers currently being published omit anything which isn't totally rigorous, even if (especially if?) it might make things clearer. I suppose this makes sense if you want to keep your papers short, which was of course important when papers were printed on, well, paper -- but these sorts of less rigorous things could certainly be made available in some sort of supplement to one's paper, published on a web page, for example.
Finally, given how many areas Euler has touched, sometimes I've joked that we should just rename mathematics "Euler". Somewhat more seriously, though -- I wonder to what extent one can tell whether someone is a mathematician by writing the word "Euler" on a piece of paper and asking them to pronounce it. (Isaac Asimov, who was trained as a chemist, suggested that "unionized" played the same role for chemists. Chemists pronounce it in four syllables and think it refers to molecules which have the same number of protons as electrons and thus a neutral charge; non-chemists pronounce it in three syllables and think it refers to organized labor.)
(My apologies for the awkwardness of the notation here. There seem to be ways to write out mathematics in blog posts -- either using LaTeX or MathML -- but they're not necessary for the notation I want to use here and seem like they'd take a little fooling around to get to work. And I don't even know MathML, although I wouldn't mind learning it.)
11 July 2007
madonna-whore complex
The median number of lifetime sexual partners for women is 3.7, for men 6.8, according to this study by the CDC which I read about at blogadilla blogadilla. The information was collected by computer, because, as is pointed out at blogadilla, "your average co-ed isn’t going to admit to some researcher that she had sex with 23 guys named Biff last Spring Break."
This seems a bit suspicious to me, because if you ask all the men in the world how many sexual partners they have had, and then you ask all the women the same question, the numbers should be approximately the same. So the mean number of sexual partners should be the same for men and for women. There are a few things that could make the means be different, even if both people involved agree they had sex:
This seems a bit suspicious to me, because if you ask all the men in the world how many sexual partners they have had, and then you ask all the women the same question, the numbers should be approximately the same. So the mean number of sexual partners should be the same for men and for women. There are a few things that could make the means be different, even if both people involved agree they had sex:
- Not all sex acts involve one man and one woman. Anecdotal evidence suggests there are more gay men than lesbians. But anecdotal evidence also suggests that there are more straight-identified women who have had sex with other women than straight-identified men who have had sex with other men. So I'm not sure which way this effect should go.
- If one partner in a sex act (that already happened) is dead and the other isn't, that'll skew things. But women live longer than men and the woman is probably on average the younger partner in a sex act, so there are probably more couples where the man has died than the woman, lowering the average for men.
- If two people engage in acts of physical intimacy, one of them might say that they've "had sex" but the other wouldn't. I'm not sure which gender is likely to do what here.
- Since this relies on self-reporting, we have to remember that people might forget who they've had sex with.
is one-fifth of New Jersey covered in lawns?
There's an old myth that one-fifth of the state of New Jersey is covered in people's lawns.
It certainly seems to be true if you drive around New Jersey for a while -- and I grew up there, so I feel like I can comment on this -- but then you start to break it down. The state of New Jersey has 7,425 square miles of land -- or 4,752,000 acres -- and has 8,414,350 people (as of the 2000 census). That means there's 0.56 acres of land for each person, or two and a quarter acres for the typical family of four. I would guess that most families that live on lots that large don't have the entire lot as a lawn, because who wants to do all that mowing?
And there's a map of how much of the U.S. is covered in lawns, from NASA. If you're at all familiar with how population is layed out in the United States you will not be surprised. The darkest green areas are, not surprisingly, the suburbs.
Lawn density seems to correlate pretty strongly with population density when population density is low, but when population density gets above a certain threshold lawn density starts to drop off; see, for example, this Wikipedia map of the population density of New Jersey. This makes sense -- people living at, say, 100 a square mile don't have lawns which are ten times as big as people living at 1,000 a square mile, because they don't want to maintain them. But people living at 10,000 a square mile have less lawn acreage per square mile even though there are ten times as many of them, because 10,000 a square mile is a city like Philadelphia or Boston where there's just no room for lawns.
This map also doesn't look all that different from this picture of the United States at night. Conclusion: the people who light up the skies are the ones with lawns.
Each different color on the map symbolizes a different density of lawn. If I knew how, I'd convert those colors back to the densities and average them out over the entire state of New Jersey and put this question to rest once and for all. (For all I know, someone at NASA has already done that.)
If you told me Long Island was one-fifth lawns, I'd believe you -- the whole island is mostly suburban.
But the state of New Jersey? I'm not sure. Lawns are a suburban phenomenon. Most of the Philadelphia suburbs are in Pennsylvania, not New Jersey. And a lot of the New York suburbs are just too dense to support a density of lawns much above one-fourth or so.
What's really shocking is that, say, the Phoenix or Las Vegas areas don't look all that different, in this map, from any other urban area. People, grass was not meant to grow in the desert. The reason that we have lawns is because in England it is not so hard to grow them, because their weather is for the most part wet and cool. But we don't live in England. (To my readers from England: well, I don't live in England. More importantly, the people of the desert Southwest don't live in England.)
(From a comment to this entry at strange maps.)
It certainly seems to be true if you drive around New Jersey for a while -- and I grew up there, so I feel like I can comment on this -- but then you start to break it down. The state of New Jersey has 7,425 square miles of land -- or 4,752,000 acres -- and has 8,414,350 people (as of the 2000 census). That means there's 0.56 acres of land for each person, or two and a quarter acres for the typical family of four. I would guess that most families that live on lots that large don't have the entire lot as a lawn, because who wants to do all that mowing?
And there's a map of how much of the U.S. is covered in lawns, from NASA. If you're at all familiar with how population is layed out in the United States you will not be surprised. The darkest green areas are, not surprisingly, the suburbs.
Lawn density seems to correlate pretty strongly with population density when population density is low, but when population density gets above a certain threshold lawn density starts to drop off; see, for example, this Wikipedia map of the population density of New Jersey. This makes sense -- people living at, say, 100 a square mile don't have lawns which are ten times as big as people living at 1,000 a square mile, because they don't want to maintain them. But people living at 10,000 a square mile have less lawn acreage per square mile even though there are ten times as many of them, because 10,000 a square mile is a city like Philadelphia or Boston where there's just no room for lawns.
This map also doesn't look all that different from this picture of the United States at night. Conclusion: the people who light up the skies are the ones with lawns.
Each different color on the map symbolizes a different density of lawn. If I knew how, I'd convert those colors back to the densities and average them out over the entire state of New Jersey and put this question to rest once and for all. (For all I know, someone at NASA has already done that.)
If you told me Long Island was one-fifth lawns, I'd believe you -- the whole island is mostly suburban.
But the state of New Jersey? I'm not sure. Lawns are a suburban phenomenon. Most of the Philadelphia suburbs are in Pennsylvania, not New Jersey. And a lot of the New York suburbs are just too dense to support a density of lawns much above one-fourth or so.
What's really shocking is that, say, the Phoenix or Las Vegas areas don't look all that different, in this map, from any other urban area. People, grass was not meant to grow in the desert. The reason that we have lawns is because in England it is not so hard to grow them, because their weather is for the most part wet and cool. But we don't live in England. (To my readers from England: well, I don't live in England. More importantly, the people of the desert Southwest don't live in England.)
(From a comment to this entry at strange maps.)
Go west, young man?
From Strange Maps: Single Guys Live in LA, Single Girls in NYC. This is a map from National Geographic showing metropolitan areas in the US by whether they have more single men or single women. The LA area has the largest plurality of single men -- the number of single men minus the number of single women -- and the New York area has the largest plurality of single women.
I think it might make more sense to show this in terms of percentages, not absolute numbers of people. (If I had their data and the time, I'd do it. I think the data is available from the Census Bureau.)
There is a similar map showing the sex ratio in each county of the US, which shows some of the same trends. The Northeast doesn't look like as much of an outlier on that map, though, probably because the Northeast has some very large urban areas. New York, Washington, Philadelphia, and Boston are #1, 4, 6, 7 respectively. But this isn't exactly what I'm looking for, because it counts children and married people. (Married couples almost always bring the sex ratio back towards 1:1, since except in Massachusetts they consist of one male and one female.)
I wouldn't be surprised to learn that states with more males have faster-growing economies, because places where the economy is growing faster probably have more risky ventures going on and risk tends to attract males more than females. The two cities out of the largest ten that have the largest proportion of females are Philadelphia and Detroit, which seems to support my idea.
Strange Maps also features a lot of other strange maps: the U.S. divided into a dozen or so smaller nations and the Antipodes Map, which visually illustrates that for most points on land, the point on the other side of the earth is in the sea. If you drill straight through the center of the Earth, you won't come out in China. Unless you're from Chile or Argentina, that is -- which are both countries from which I've had exactly one page view.
I think it might make more sense to show this in terms of percentages, not absolute numbers of people. (If I had their data and the time, I'd do it. I think the data is available from the Census Bureau.)
There is a similar map showing the sex ratio in each county of the US, which shows some of the same trends. The Northeast doesn't look like as much of an outlier on that map, though, probably because the Northeast has some very large urban areas. New York, Washington, Philadelphia, and Boston are #1, 4, 6, 7 respectively. But this isn't exactly what I'm looking for, because it counts children and married people. (Married couples almost always bring the sex ratio back towards 1:1, since except in Massachusetts they consist of one male and one female.)
I wouldn't be surprised to learn that states with more males have faster-growing economies, because places where the economy is growing faster probably have more risky ventures going on and risk tends to attract males more than females. The two cities out of the largest ten that have the largest proportion of females are Philadelphia and Detroit, which seems to support my idea.
Strange Maps also features a lot of other strange maps: the U.S. divided into a dozen or so smaller nations and the Antipodes Map, which visually illustrates that for most points on land, the point on the other side of the earth is in the sea. If you drill straight through the center of the Earth, you won't come out in China. Unless you're from Chile or Argentina, that is -- which are both countries from which I've had exactly one page view.
10 July 2007
most common line score?
Major League Baseball is doing what they're calling the scoreboard challenge for the All-Star Game tonight. They challenge people to guess the number of runs that will be scored in each half inning, as well as the final number of hits and errors for each team.
Ignoring the hits and errors for a moment: what do you think the most common set of scores in each half inning is?
My guess is that it's no score in each half inning except for a single run in the bottom of the first. Why? Well, in any randomly chosen half-inning, 0 is the most common number of runs scored. So I'd go for the all-zeroes line score -- but then the game wouldn't end! So I begrudgingly allow a single run to score.
Why does that run score in the bottom of the first, you ask? The home team is more likely to score than the visiting team, so it should be in the bottom of some inning. And in general, teams score more in the first inning than any other inning, because batting orders are set up so that the best hitters bat first, so I'll put the run in the first inning. (The second inning, by the way, is historically the lowest-scoring inning.)
This seems a bit counterintuitive, because if I wake up tomorrow (the game starts at eight and will probably last forever because All-Star games are full of substitutions, so I'm not sure if I'll stay awake for the whole thing) and see that the score was
I'd surely say "oh my god, what happened?" On average a baseball team scores five runs or so, so something like the 2005 All-Star game line score
seems a lot more likely. And in fact, the National League scoring two runs in each of two inning and one in a third inning, and the American League scoring two runs in each of three innings and one in a fourth inning, probably is more likely than the 1-0 game I invented -- but only because we didn't specify which innings.
The people who designed this contest seem to be aware of this, because the rules specify that they give people 5 points for each correct 0 or 1; 10 points for each correct 2 or 3; and 20 for each correct >= 4.
Now, how often does a team not score at all in a half-inning? Looking at Sunday's games, 264 half-innings of Major League Baseball were played; 69 had at least one run. On Saturday, it was 67 out of 301. On Friday, 97 out of 290. (Friday was a high-scoring day.) That's 233 out of 855 for the weekend. In general, we see that about 73% of innings have no runs. Similarly, there were 49 one-run innings on Friday, 32 on Saturday, and 33 on Sunday, so 114 out of 855, or 13%.
So if we assume that all innings are independent, then the probability of a given game having the line score I gave is (.73)17(.13) or about 0.00060; there should be, on average, one game like this out of every 1,600 or so. There are 2,430 games played each season (162 games per team, times thirty teams, divided by two), so we expect one and a half games with that line score per season.
Except the actual number should be a bit more, because:
But if you actually go through the 38 1-0 games that occurred last season, the single runs scored as follows:
and there were no 1-0 games where the only run scored in the top or bottom of the first inning -- this sort of thing can happen with rare events. I'm kind of curious if this has ever occurred -- for the reasons I outlined above, I suspect it has -- but not curious enough to go digging back any further.
If you'd like to use what I've said to enter in the contest, it's here -- but hurry, the deadline is 7:59 PM Eastern.
edit, 10:04 pm: Four innings are done. The National League is up 1-0... and they scored that run in the bottom of the first.
Ignoring the hits and errors for a moment: what do you think the most common set of scores in each half inning is?
My guess is that it's no score in each half inning except for a single run in the bottom of the first. Why? Well, in any randomly chosen half-inning, 0 is the most common number of runs scored. So I'd go for the all-zeroes line score -- but then the game wouldn't end! So I begrudgingly allow a single run to score.
Why does that run score in the bottom of the first, you ask? The home team is more likely to score than the visiting team, so it should be in the bottom of some inning. And in general, teams score more in the first inning than any other inning, because batting orders are set up so that the best hitters bat first, so I'll put the run in the first inning. (The second inning, by the way, is historically the lowest-scoring inning.)
This seems a bit counterintuitive, because if I wake up tomorrow (the game starts at eight and will probably last forever because All-Star games are full of substitutions, so I'm not sure if I'll stay awake for the whole thing) and see that the score was
American: 000 000 000
National: 100 000 00x
I'd surely say "oh my god, what happened?" On average a baseball team scores five runs or so, so something like the 2005 All-Star game line score
National: 000 000 212
American: 012 202 00x
seems a lot more likely. And in fact, the National League scoring two runs in each of two inning and one in a third inning, and the American League scoring two runs in each of three innings and one in a fourth inning, probably is more likely than the 1-0 game I invented -- but only because we didn't specify which innings.
The people who designed this contest seem to be aware of this, because the rules specify that they give people 5 points for each correct 0 or 1; 10 points for each correct 2 or 3; and 20 for each correct >= 4.
Now, how often does a team not score at all in a half-inning? Looking at Sunday's games, 264 half-innings of Major League Baseball were played; 69 had at least one run. On Saturday, it was 67 out of 301. On Friday, 97 out of 290. (Friday was a high-scoring day.) That's 233 out of 855 for the weekend. In general, we see that about 73% of innings have no runs. Similarly, there were 49 one-run innings on Friday, 32 on Saturday, and 33 on Sunday, so 114 out of 855, or 13%.
So if we assume that all innings are independent, then the probability of a given game having the line score I gave is (.73)17(.13) or about 0.00060; there should be, on average, one game like this out of every 1,600 or so. There are 2,430 games played each season (162 games per team, times thirty teams, divided by two), so we expect one and a half games with that line score per season.
Except the actual number should be a bit more, because:
- the innings of a game aren't independent. In particular, most of them are pitched by the same pitcher
- like I said before, if you're ever going to score, you're going to do it in the bottom of the first.
But if you actually go through the 38 1-0 games that occurred last season, the single runs scored as follows:
visitors : 011 343 300 000 0
home team: 012 424 222 210 1
and there were no 1-0 games where the only run scored in the top or bottom of the first inning -- this sort of thing can happen with rare events. I'm kind of curious if this has ever occurred -- for the reasons I outlined above, I suspect it has -- but not curious enough to go digging back any further.
If you'd like to use what I've said to enter in the contest, it's here -- but hurry, the deadline is 7:59 PM Eastern.
edit, 10:04 pm: Four innings are done. The National League is up 1-0... and they scored that run in the bottom of the first.
"Typewriter Trivia"
I do too many crosswords.
The New York Sun runs an excellent crossword, which yesterday was entitled "Typewriter Trivia" and featured the following four long entries, here with their clues:
Disposition to credulity (and the longest common word that alternates typing hands): ANTISKEPTICISM
Violet variety (and the longest common word that uses just the right typing hand): JOHNNYJUMPUP
Seesaw (and the longest common word that uses just the top typewriter row): TEETERTOTTER
Knitted garments for women (and the longest common word that uses just the left typing hand): SWEATERDRESSES
If you want to read more about crosswords, check out Amy Reynaldo's blog Diary of a Crossword Fiend -- the link is to the entry on the crosswords which were published yesterday. This particular entry also mentions her book How to Conquer the New York Times Crossword Puzzle: Tips, Tricks and Techniques to Master America's Favorite Puzzle
, which comes out today.
Wikipedia, which has a lot of silly lists like this, tells us that TESSERADECADES and AFTERCATARACTS are also typeable entirely with the left hand, though they're less common. TETRASTEARATES also has this property, according to A Collection of Word Oddities and Trivia. To someone who does too many crosswords, it's also recognizable as a word that would often be found up against the right or bottom edge of a crossword, because it consists of letters that occur often at the end of words. (ASSERTS, ASSESSES, and so on are common in those positions.)
The longest word entirely typeable with the middle row is SHAKALSHAS, which is shorter than the others and also more obscure. There are no words which are entirely typeable with the bottom row of the standard keyboary, since it has no vowels. (The
What I'm led to wonder is, do we expect such words to be long? Longer words should be possible if we have larger sets of letters to work with. Georges Perec once wrote a novel called La Disparition which doesn't contain the letter "e" (and in French, no less, where this should be harder than in English!) and also Les Revenentes which contains no a, i, o, or u. The first of these was "translated" into English as A Void, although I don't know enough to know if this can really be called a "translation". This is considered noteworthy. But if someone wrote a book without a Q in it, nobody would care!
Letter frequencies can be found here, without regard for the frequency of the word in ordinary text. We can compute that 48.03% of letters come from the top row; 32.49% from the middle row; 19.74% from the bottom row. So we expect that it will be easiest to form words which come entirely from the top row, then the middle row, then the bottom row; this is in fact the case. Similarly, 61.27% of letters are drawn from the left hand; we can form even longer words with the left hand (14 letters) than with the top row (12 letters).
As for why we can form such long words with alternating hands? The Dvorak keyboard, which is a widely used alternate to the standard QWERTY keyboard, is set up so that letters which often occur consecutively are typed with opposite hands. It wouldn't surprise me if QWERTY uses that same principle, even if it's not consciously incorporated into the design.
(An earlier version of this post had some broken links. They should be fixed now.)
The New York Sun runs an excellent crossword, which yesterday was entitled "Typewriter Trivia" and featured the following four long entries, here with their clues:
Disposition to credulity (and the longest common word that alternates typing hands): ANTISKEPTICISM
Violet variety (and the longest common word that uses just the right typing hand): JOHNNYJUMPUP
Seesaw (and the longest common word that uses just the top typewriter row): TEETERTOTTER
Knitted garments for women (and the longest common word that uses just the left typing hand): SWEATERDRESSES
If you want to read more about crosswords, check out Amy Reynaldo's blog Diary of a Crossword Fiend -- the link is to the entry on the crosswords which were published yesterday. This particular entry also mentions her book How to Conquer the New York Times Crossword Puzzle: Tips, Tricks and Techniques to Master America's Favorite Puzzle
, which comes out today.Wikipedia, which has a lot of silly lists like this, tells us that TESSERADECADES and AFTERCATARACTS are also typeable entirely with the left hand, though they're less common. TETRASTEARATES also has this property, according to A Collection of Word Oddities and Trivia. To someone who does too many crosswords, it's also recognizable as a word that would often be found up against the right or bottom edge of a crossword, because it consists of letters that occur often at the end of words. (ASSERTS, ASSESSES, and so on are common in those positions.)
The longest word entirely typeable with the middle row is SHAKALSHAS, which is shorter than the others and also more obscure. There are no words which are entirely typeable with the bottom row of the standard keyboary, since it has no vowels. (The
What I'm led to wonder is, do we expect such words to be long? Longer words should be possible if we have larger sets of letters to work with. Georges Perec once wrote a novel called La Disparition which doesn't contain the letter "e" (and in French, no less, where this should be harder than in English!) and also Les Revenentes which contains no a, i, o, or u. The first of these was "translated" into English as A Void, although I don't know enough to know if this can really be called a "translation". This is considered noteworthy. But if someone wrote a book without a Q in it, nobody would care!
Letter frequencies can be found here, without regard for the frequency of the word in ordinary text. We can compute that 48.03% of letters come from the top row; 32.49% from the middle row; 19.74% from the bottom row. So we expect that it will be easiest to form words which come entirely from the top row, then the middle row, then the bottom row; this is in fact the case. Similarly, 61.27% of letters are drawn from the left hand; we can form even longer words with the left hand (14 letters) than with the top row (12 letters).
As for why we can form such long words with alternating hands? The Dvorak keyboard, which is a widely used alternate to the standard QWERTY keyboard, is set up so that letters which often occur consecutively are typed with opposite hands. It wouldn't surprise me if QWERTY uses that same principle, even if it's not consciously incorporated into the design.
(An earlier version of this post had some broken links. They should be fixed now.)
compression = artificial intelligence?: the Hutter Prize
From Slashdot: Alexander Ratushnyak has won the second Hutter prize. This is a prize for compressing the first 100 megabytes of Wikipedia to a small size. The size that he achieved is 16,481,655 bytes; in March of 2006, before this prize was announced, the best possible was 18,324,887 bytes.
Hutter claims that "being able to compress well is closely related to acting intelligently" -- this is based on the thesis that the way to compress data well is to have a good idea of what's coming next, and the way to do that is to understand the structure of the data. So far, so good. But does it follow that understanding the structure of the data requires intelligence on the part of the compression algorithm? I don't think so. The MP3 algorithm, for example, compresses music for the most part by relying on psychological information about how humans perceive sound, and it required intelligence on the part of the inventor -- but I wouldn't ascribe intelligence to the algorithm.
Supposedly there is a proof that "ideal lossless compression" implies passing the Turing Test: the e-mail there by Matt Mahoney states
This may be true -- I'm not competent to evaluate the proof. But it does not follow that a compression method which is 1% short of ideal lossless compression is 1% away from passing the Turing test. And we don't know what the ideal even is! It is possible to estimate the entropy of natural language, but the estimates seem to be good to, say, one decimal place.
Dan has pointed out at my post secret messages in human DNA? that general-purpose compressors don't work in all data. If you gzip the human genome, it gets bigger. In fact, it's not possible to write a program that will losslessly compress all input strings, for the following reason: say we can write a program which losslessly compresses all strings of length up to N bits. There are 2N+1-1 such strings, but only 2N-1 strings of length N-1 bits or less; therefore there will be collisions among the strings of length N-1 or less, and we won't be able to properly uncompress the data.
Hutter claims that "being able to compress well is closely related to acting intelligently" -- this is based on the thesis that the way to compress data well is to have a good idea of what's coming next, and the way to do that is to understand the structure of the data. So far, so good. But does it follow that understanding the structure of the data requires intelligence on the part of the compression algorithm? I don't think so. The MP3 algorithm, for example, compresses music for the most part by relying on psychological information about how humans perceive sound, and it required intelligence on the part of the inventor -- but I wouldn't ascribe intelligence to the algorithm.
Supposedly there is a proof that "ideal lossless compression" implies passing the Turing Test: the e-mail there by Matt Mahoney states
The second problem remains: ideal lossy compression does not imply passing the Turing test. For lossless compression, it can be proven that it does. Let p(s) be the (unknown) probability that s will be the prefix of a text dialog. Then a machine that can compute p(s) exactly is able to generate response A to question Q with the distribution p(QA)/p(Q) which is indistinguishable from human. The same model minimizes the compressed size, E[log 1/p(s)].
This may be true -- I'm not competent to evaluate the proof. But it does not follow that a compression method which is 1% short of ideal lossless compression is 1% away from passing the Turing test. And we don't know what the ideal even is! It is possible to estimate the entropy of natural language, but the estimates seem to be good to, say, one decimal place.
Dan has pointed out at my post secret messages in human DNA? that general-purpose compressors don't work in all data. If you gzip the human genome, it gets bigger. In fact, it's not possible to write a program that will losslessly compress all input strings, for the following reason: say we can write a program which losslessly compresses all strings of length up to N bits. There are 2N+1-1 such strings, but only 2N-1 strings of length N-1 bits or less; therefore there will be collisions among the strings of length N-1 or less, and we won't be able to properly uncompress the data.
Are you afraid of math?
Are you afraid of math? (Probably not, since you're reading this blog.)
But if you are, you can buy on eBay Five Books For Those Who Suffer From "Math Phobia", at what look to be the middle school level -- and the seller says none of them have ever been used. Perhaps ey is so mathphobic that ey never bothered to even open the books?
A question that I've long wondered about is why a fear of math seems to be a lot more common than, say, a fear of reading. It seems difficult to get a good answer to this question, but I suspect I'll have thoughts on it in the future which will be posted in this space.
But if you are, you can buy on eBay Five Books For Those Who Suffer From "Math Phobia", at what look to be the middle school level -- and the seller says none of them have ever been used. Perhaps ey is so mathphobic that ey never bothered to even open the books?
A question that I've long wondered about is why a fear of math seems to be a lot more common than, say, a fear of reading. It seems difficult to get a good answer to this question, but I suspect I'll have thoughts on it in the future which will be posted in this space.
09 July 2007
lucky babies
Stephen at Freakonomics asks:
I wonder if there are any. You'd expect 11,000/1440 = 7.5 babies to be born in that minute, and 7.5 in the corresponding p.m. minute, for a total of fifteen. About two percent of babies weigh between 7 lbs, 6.5 oz and 7 lbs, 7.5 oz at birth; I'm getting this from this analysis of Norwegian birth weights which is all I could find quickly. (Seven percent of babies weigh between 3350 grams and 3450 grams at birth; one ounce is very nearly two-sevenths of one hundred grams.) So the expected number of 7 lb, 7 oz. babies born at 7:07 (a.m. or p.m., local time) on 7/7/07 is about two percent of fifteen, or 0.3. I'd guess that the number of babies born during any minute is Poisson-distributed; the probability there are no such babies is thus e-0.3, or about 74%.
Then again, as has been pointed out, if you had a seven-pound, six-ounce baby born at 7:08, maybe you'd make everything be 7s just for the hell of it. Hospital staff aren't above this sort of thing. I was born prematurely and weighed five pounds, eight ounces at birth, but lost weight once I was born; the hospital had a rule that babies under five pounds couldn't leave. On Christmas Eve I weighed a bit under five pounds; apparently the records show five pounds exactly, because the nurse felt that I should be home for Christmas. (Given who gets stuck with working at a hospital on Christmas Day, this might have even made sense from a medical point of view.)
Of course, the U.S. isn't the entire world. We're about one-twentieth of the world's population; very crudely just multiplying that 0.3 by twenty gives six. (This of course doesn't take into account the different birth rates or distributions of birth weights in different countries.) Then the probability that there are no such babies is about e-6, or one in 400 (by the way, knowing e3 is very nearly 20 is useful), and even without rounding there probably do exist such supremely "lucky" babies.
What I want to hear about is the 7 lbs.-7 oz. kid who was born at 7:07 a.m. on 7/7/07. Any leads? She will probably grow up to be a poker champ.
I wonder if there are any. You'd expect 11,000/1440 = 7.5 babies to be born in that minute, and 7.5 in the corresponding p.m. minute, for a total of fifteen. About two percent of babies weigh between 7 lbs, 6.5 oz and 7 lbs, 7.5 oz at birth; I'm getting this from this analysis of Norwegian birth weights which is all I could find quickly. (Seven percent of babies weigh between 3350 grams and 3450 grams at birth; one ounce is very nearly two-sevenths of one hundred grams.) So the expected number of 7 lb, 7 oz. babies born at 7:07 (a.m. or p.m., local time) on 7/7/07 is about two percent of fifteen, or 0.3. I'd guess that the number of babies born during any minute is Poisson-distributed; the probability there are no such babies is thus e-0.3, or about 74%.
Then again, as has been pointed out, if you had a seven-pound, six-ounce baby born at 7:08, maybe you'd make everything be 7s just for the hell of it. Hospital staff aren't above this sort of thing. I was born prematurely and weighed five pounds, eight ounces at birth, but lost weight once I was born; the hospital had a rule that babies under five pounds couldn't leave. On Christmas Eve I weighed a bit under five pounds; apparently the records show five pounds exactly, because the nurse felt that I should be home for Christmas. (Given who gets stuck with working at a hospital on Christmas Day, this might have even made sense from a medical point of view.)
Of course, the U.S. isn't the entire world. We're about one-twentieth of the world's population; very crudely just multiplying that 0.3 by twenty gives six. (This of course doesn't take into account the different birth rates or distributions of birth weights in different countries.) Then the probability that there are no such babies is about e-6, or one in 400 (by the way, knowing e3 is very nearly 20 is useful), and even without rounding there probably do exist such supremely "lucky" babies.
what's the heat index, anyway?
It's never really been clear to me what the "heat index" means.
Today in Philadelphia, at 4 pm, it's supposed to hit 95 degrees with 34% humidity; that corresponds, according to weather.com, to a heat index of 98. Tomorrow we expect 95 degrees and 43% relative humidity, for a heat index of 102.
What I've noticed is that the heat index here is almost always higher than the actual temperature. It'll get up to 95 on Tuesday, and someone will say "it `feels like' 102 degrees". No! It feels like 95. This is what 95 feels like around here. Sure, it might feel hotter than 95 degrees in the desert, but I've never been to the desert.
There are various formulas for the heat index. I suspect the second of the three given at Wikipedia is the "most accurate" in some sense, because it involves exponentiation of the reciprocal temperature, which is something which arises often in statistical mechanics; the other two are probably just polynomial approximations of it, which are easier to calculate, but ease of calculation is not so important. The various sources online seem a bit vague, though.
The following document from the National Weather Service states that the third approximation given at Wikipedia is, indeed, a polynomial approximation of the "true" heat index, which apparently depends on a fairly complicated biological model. In the end it probably just makes sense to resort to tables.
What surprises me is that none of the online formulas take into account wind. (The NWS document says that the wind is assumed to be a constant 5 knots in the model.) This seems silly to me. When I'm inside I can turn on a fan and be cooler than I would if the fan weren't on, because the fan dissipates the hot air around my body; shouldn't the same be true outside? (Michael Bluejay taught me this about fans.)
I think that the heat index is misleading, because it makes people think it's hotter than it actually is. I'd actually support replacing it with a number with an arbitrary scale, say zero to ten. If someone told me that tomorrow's going to be a nine on that scale, I'd know I don't want to go outside.
Also, weather.com has two forecasts that I look at regularly -- their ten-day forecast and their hour-by-hour forecast. The ten-day forecast says the high will be 97 today and tomorrow; the hourly forecast only goes up to 95. I suspect the reason is that at any given hour the expected temperature is indeed 95 degrees and the exact time that it will reach 97 is not known. Similarly, the forecast low Tuesday morning according to the ten-day forecast is 77, and the hourly only gets down to 78; the forecast low Wednesday morning is 76 but the hourly only gets down to 77.
Today in Philadelphia, at 4 pm, it's supposed to hit 95 degrees with 34% humidity; that corresponds, according to weather.com, to a heat index of 98. Tomorrow we expect 95 degrees and 43% relative humidity, for a heat index of 102.
What I've noticed is that the heat index here is almost always higher than the actual temperature. It'll get up to 95 on Tuesday, and someone will say "it `feels like' 102 degrees". No! It feels like 95. This is what 95 feels like around here. Sure, it might feel hotter than 95 degrees in the desert, but I've never been to the desert.
There are various formulas for the heat index. I suspect the second of the three given at Wikipedia is the "most accurate" in some sense, because it involves exponentiation of the reciprocal temperature, which is something which arises often in statistical mechanics; the other two are probably just polynomial approximations of it, which are easier to calculate, but ease of calculation is not so important. The various sources online seem a bit vague, though.
The following document from the National Weather Service states that the third approximation given at Wikipedia is, indeed, a polynomial approximation of the "true" heat index, which apparently depends on a fairly complicated biological model. In the end it probably just makes sense to resort to tables.
What surprises me is that none of the online formulas take into account wind. (The NWS document says that the wind is assumed to be a constant 5 knots in the model.) This seems silly to me. When I'm inside I can turn on a fan and be cooler than I would if the fan weren't on, because the fan dissipates the hot air around my body; shouldn't the same be true outside? (Michael Bluejay taught me this about fans.)
I think that the heat index is misleading, because it makes people think it's hotter than it actually is. I'd actually support replacing it with a number with an arbitrary scale, say zero to ten. If someone told me that tomorrow's going to be a nine on that scale, I'd know I don't want to go outside.
Also, weather.com has two forecasts that I look at regularly -- their ten-day forecast and their hour-by-hour forecast. The ten-day forecast says the high will be 97 today and tomorrow; the hourly forecast only goes up to 95. I suspect the reason is that at any given hour the expected temperature is indeed 95 degrees and the exact time that it will reach 97 is not known. Similarly, the forecast low Tuesday morning according to the ten-day forecast is 77, and the hourly only gets down to 78; the forecast low Wednesday morning is 76 but the hourly only gets down to 77.
08 July 2007
rounders and accumulators, satisficers and maximizers
In the Dilbert blog, Scott Adams makes a distinction between two types of people, who he calls "rounders" and "accumulators":
This reminded me of another distinction I've read about. Barry Schwartz, author of a book entitled The Paradox of Choice: Why More Is Less
, draws a distinction between "satisficers" and "maximizers". Satisficers are the people who, when they want, say, a computer, will walk into a store and buy the first computer they think is "good enough"; maximizers are the ones who will spend ridiculous amounts of time worrying about which is the "best" computer. In the end they might save $100 (or get $100 more of computing power for the same price) by doing so, but it's not worth the stress it causes them. This is becoming more and more of a problem because there are so many consumer products out there. Psychologically, maximizing turns out to be quite unhealthy. This isn't exactly the same thing as the rounder/accumulator distinction, but they're similar. In both cases, there's a type of person who is more able to shrug problems off and as a result has better mental health; the difference is in exactly how that shrugging off of problems comes about.
How can we model this mathematically, though? One of the threads that runs through Schwartz's book is that in some sense satisficers really are maximizers -- they just value their time and energy more, so settling for a "good enough" thing that isn't the best possible is maximizing, for them. The question becomes -- at what point does the time one spends examining one more alternative outweigh the possibility that that one more alternative might be "the one"?
Say that the quality of some pool of objects are given by random numbers uniformly distributed from 0 to 1. (This is unrealistic, I know, but it makes the computation easier.) It can be shown that if you choose n such numbers at random, the expected value of the maximum is n/(n+1) -- if you pick a single number its expected value is 1/2, the expected value of the larger of two such numbers is 2/3, and so on. Notice that we gain quite a bit in going from one object to two -- we gain 1/6. In going from n objects to n+1 we gain 1/(n+1)(n+2), which shrinks pretty quickly.
Now let's say that the cost of "picking" a single object and determining which random number it corresponds to is k. Then if we pick out n objects and assess their quality, we expect to have one which has quality n/(n+1); but we expended kn in finding it, so overall we win by n/(n+1) - kn. We want to maximize this. It turns out this is maximized when n = 1 + 1/√k.
What is k be, numerically? Roughly, it's the reciprocal of the potential variation in the "quality" of your objects divided by the cost of researching one well enough to know what it costs. For example, say we're looking at computers which are priced at $1000, and you think they're "really" worth somewhere between $900 and $1100. Furthermore, let's say it takes you fifteen minutes to determine what you think a particular computer is "really" worth, and you value your time at $20 an hour. Then k should be on the order of $5/$200, or 1/40. You could probably argue for other values but they wouldn't differ from this by more than a factor of two or so. But if computers varied more, k would be bigger -- say their true values varied between $500 and $1500, then k would be five times as large. Or say you were buying a $20,000 car instead of a $1,000 computer, and again the "true value" could be as much as ten percent off, between $18,000 and $22,000; now k should be twenty times as large. The scaling is much more important than the actual number.
So if we're looking at more expensive things, we should look harder -- but not that much harder. If the things we're looking at are a hundred times as expensive (or if we're buying a hundred times as many of them -- say we're buying computers for a business, not for ourselves), we should spend ten times as much time looking.
The uniform distribution probably isn't the right one to use here. But that's not really the problem; the real problem is that we don't know the distribution in advance. And the job of the advertising industry is basically to convince us that the distribution isn't what we thought it was -- that there is a product that's so great that we should spend a day in line waiting for it, even if you're the mayor of the country'sfifth sixth-largest city where six murders in a day is only barely front-page news.
ROUNDERS: This group rounds things off. A problem that’s a two on a scale of one to ten gets rounded to zero. If a rounder has five problems that are all about a two on a scale of one to ten, he’ll tell you he has no problems.
ACCUMULATORS: Accumulators add up all the little problems until they equal one big problem. If an accumulator has five problems that are each a two on a scale of one to ten, that feels like having one problem that’s a ten.
This reminded me of another distinction I've read about. Barry Schwartz, author of a book entitled The Paradox of Choice: Why More Is Less
, draws a distinction between "satisficers" and "maximizers". Satisficers are the people who, when they want, say, a computer, will walk into a store and buy the first computer they think is "good enough"; maximizers are the ones who will spend ridiculous amounts of time worrying about which is the "best" computer. In the end they might save $100 (or get $100 more of computing power for the same price) by doing so, but it's not worth the stress it causes them. This is becoming more and more of a problem because there are so many consumer products out there. Psychologically, maximizing turns out to be quite unhealthy. This isn't exactly the same thing as the rounder/accumulator distinction, but they're similar. In both cases, there's a type of person who is more able to shrug problems off and as a result has better mental health; the difference is in exactly how that shrugging off of problems comes about.How can we model this mathematically, though? One of the threads that runs through Schwartz's book is that in some sense satisficers really are maximizers -- they just value their time and energy more, so settling for a "good enough" thing that isn't the best possible is maximizing, for them. The question becomes -- at what point does the time one spends examining one more alternative outweigh the possibility that that one more alternative might be "the one"?
Say that the quality of some pool of objects are given by random numbers uniformly distributed from 0 to 1. (This is unrealistic, I know, but it makes the computation easier.) It can be shown that if you choose n such numbers at random, the expected value of the maximum is n/(n+1) -- if you pick a single number its expected value is 1/2, the expected value of the larger of two such numbers is 2/3, and so on. Notice that we gain quite a bit in going from one object to two -- we gain 1/6. In going from n objects to n+1 we gain 1/(n+1)(n+2), which shrinks pretty quickly.
Now let's say that the cost of "picking" a single object and determining which random number it corresponds to is k. Then if we pick out n objects and assess their quality, we expect to have one which has quality n/(n+1); but we expended kn in finding it, so overall we win by n/(n+1) - kn. We want to maximize this. It turns out this is maximized when n = 1 + 1/√k.
What is k be, numerically? Roughly, it's the reciprocal of the potential variation in the "quality" of your objects divided by the cost of researching one well enough to know what it costs. For example, say we're looking at computers which are priced at $1000, and you think they're "really" worth somewhere between $900 and $1100. Furthermore, let's say it takes you fifteen minutes to determine what you think a particular computer is "really" worth, and you value your time at $20 an hour. Then k should be on the order of $5/$200, or 1/40. You could probably argue for other values but they wouldn't differ from this by more than a factor of two or so. But if computers varied more, k would be bigger -- say their true values varied between $500 and $1500, then k would be five times as large. Or say you were buying a $20,000 car instead of a $1,000 computer, and again the "true value" could be as much as ten percent off, between $18,000 and $22,000; now k should be twenty times as large. The scaling is much more important than the actual number.
So if we're looking at more expensive things, we should look harder -- but not that much harder. If the things we're looking at are a hundred times as expensive (or if we're buying a hundred times as many of them -- say we're buying computers for a business, not for ourselves), we should spend ten times as much time looking.
The uniform distribution probably isn't the right one to use here. But that's not really the problem; the real problem is that we don't know the distribution in advance. And the job of the advertising industry is basically to convince us that the distribution isn't what we thought it was -- that there is a product that's so great that we should spend a day in line waiting for it, even if you're the mayor of the country's
what does "losing 35 IQ points" mean?
The Gregarious Brain, in today's New York Times magazine. The article is about people who suffer from Williams syndrome, and focuses on the fact that people with Williams have trouble with abstract thought and have "exuberant gregariousness and near-normal language skills". These individuals don't have the best social skills, though, which makes one feel sorry for them -- they want to connect, but they can't.
This comes about because the part of their brains which deals with abstract thought (the dorsal part) are underdeveloped, but the parts dealing with language (the ventral part) are normally developed, because certain genes don't act during the formation of the brain.
What grabbed me, mathematically, was this:
Does this mean anything? Obviously "I. Q. points" are not something which just sits there in our brain. If my IQ is, say, 130, there aren't 130 little blobs sitting there in my brain which do my thinking for me -- or even 1.3 times as many such little blobs as the average person. (I know, you might be thinking that neurons are such "little blobs", but brain size isn't correlated very well with intelligence.) Furthermore, IQ scores are set up to have a Gaussian distribution, which I suspect is not the right thing to do. Perhaps the intelligence of individuals whose brains have developed "normally" are normally distributed, but the fact that there are a large number of disorders which "take away" IQ points makes me think there'd be a bump around, say, 60 or 70 IQ points -- a couple standard deviations below the median.
And that's only if intelligence is normally distributed to begin with. You'd expect that if intelligence were the sum of a bunch of independent effects, but I suspect there's some sort of synergy going on where "the whole is greater than the sum of its parts" -- a moderately above-average ability in, say, spatial reasoning and computational ability might make a better mathematician than someone who's really good at one of those and only average at the other. In general there are lots of complex skills which are made up of simpler skills in this way.
I suspect the central part of the distribution is approximately normal, though; the weirdness probably goes on with the very smart or the very stupid.
This comes about because the part of their brains which deals with abstract thought (the dorsal part) are underdeveloped, but the parts dealing with language (the ventral part) are normally developed, because certain genes don't act during the formation of the brain.
What grabbed me, mathematically, was this:
These deficits generally erase about 35 points from whatever I.Q. the person would have inherited without the deletion. Since the average I.Q. is 100, this leaves most people with Williams with I.Q.’s in the 60s. Though some can hold simple jobs, they require assistance managing their lives.
Does this mean anything? Obviously "I. Q. points" are not something which just sits there in our brain. If my IQ is, say, 130, there aren't 130 little blobs sitting there in my brain which do my thinking for me -- or even 1.3 times as many such little blobs as the average person. (I know, you might be thinking that neurons are such "little blobs", but brain size isn't correlated very well with intelligence.) Furthermore, IQ scores are set up to have a Gaussian distribution, which I suspect is not the right thing to do. Perhaps the intelligence of individuals whose brains have developed "normally" are normally distributed, but the fact that there are a large number of disorders which "take away" IQ points makes me think there'd be a bump around, say, 60 or 70 IQ points -- a couple standard deviations below the median.
And that's only if intelligence is normally distributed to begin with. You'd expect that if intelligence were the sum of a bunch of independent effects, but I suspect there's some sort of synergy going on where "the whole is greater than the sum of its parts" -- a moderately above-average ability in, say, spatial reasoning and computational ability might make a better mathematician than someone who's really good at one of those and only average at the other. In general there are lots of complex skills which are made up of simpler skills in this way.
I suspect the central part of the distribution is approximately normal, though; the weirdness probably goes on with the very smart or the very stupid.
07 July 2007
10000 losses, yet again
I posted a couple weeks ago a forecast of the date of the 10,000th Phillies loss, which I updated on Tuesday. My traffic-tracking software tells me that these have been my most-viewed and fourth-most-viewed pages, respectively. In particular I wondered what the chances were that I'd witness this historic event. I have tickets for July 13. The July 14 game will be televised nationally on FOX, the July 15 on ESPN; I'm certain the announcers will mention it, either as "the Phillies have just lost their 10,000th game" or "the Phillies are trying to avoid losing their 10,000th game" depending on how things play out.
Since then, the Phillies have won three and lost seven; as of the original post they'd lost 9,991 games all-time, so now they're up to 9,998 losses. They need two more for 10,000.
They play tonight, then again tomorrow, then after that not until Friday (which is when I have tickets). Who knows, I might see it.
Here are the probabilities now:
The single most likely game is now tomorrow's game; not surprisingly the Phillies have roughly a one-in-four chance of losing tonight and tomorrow afternoon and just getting this whole mess over with. There's almost a fifty-fifty shot of it coming during the Cardinals series (47.2%, to be exact); a 26.4% chance of it happening on the West Coast swing, which is when I originally thought it would happen; and an 0.6% chance of it happening it after their return from the West Coast.
By the way, current records for all teams are available at baseball-reference.com. The only team to have 10,000 wins so far are the Giants, with 10,150; the Cubs will most likely be next to reach that milestone, with 9,943. The Braves will be the next to cross the 10,000-loss line, but they're 320 short so it'll take a few years.
If you noticed that those are all National League teams, that's not a coincidence. The Phillies, Giants, Cubs, and Braves started play in 1883, 1883, 1876, and 1876 respectively; the eight original AL teams -- today's Orioles, Red Sox, White Sox, Indians, Tigers, Twins, Yankees, and A's -- all started in 1901, when that league was founded.)
The Phillies aren't the team with the lowest winning percentage, not by a long shot; they're .468 all-time. The Rangers, Rockies, and Padres are a bit worse at .467, .466, and .463; the Devil Rays are .398 all time. But they're all expansion teams, and expansion teams are historically bad the Phillies do have the worst winning percentage of the original sixteen teams. (The original NL is the Braves, Phillies, Cubs, Cardinals, Dodgers, Giants, Reds, and Pirates.) You can see online the standings of the the eight original AL teams and NL teams graphed since 1901. It won't surprise anyone to learn the Yankees are the best AL team in that time, and the Giants are only a bit less surprisingly the best NL team.
Since then, the Phillies have won three and lost seven; as of the original post they'd lost 9,991 games all-time, so now they're up to 9,998 losses. They need two more for 10,000.
They play tonight, then again tomorrow, then after that not until Friday (which is when I have tickets). Who knows, I might see it.
Here are the probabilities now:
| Jul 08 | @ Rockies | 0.257245 |
| Jul 13 | v. Cardinals | 0.197124 |
| Jul 14 | v. Cardinals | 0.157155 |
| Jul 15 | v. Cardinals | 0.118057 |
| Jul 16 | @ Dodgers | 0.126696 |
| Jul 17 | @ Dodgers | 0.071025 |
| Jul 18 | @ Dodgers | 0.037119 |
| Jul 19 | @ Padres | 0.018392 |
| Jul 20 | @ Padres | 0.009024 |
| Jul 21 | @ Padres | 0.004337 |
| Jul 22 | @ Padres | 0.002051 |
| Jul 24 | v. Nationals | 0.000582 |
| Jul 25 | v. Nationals | 0.000392 |
| Jul 26 | v. Nationals | 0.000264 |
| Jul 27 | v. Pirates | 0.000180 |
| Jul 28 | v. Pirates | 0.000120 |
| Jul 29 | v. Pirates | 0.000080 |
| Jul 30 | @ Cubs | 0.000074 |
| Jul 31 | @ Cubs | 0.000039 |
| Aug 01 | @ Cubs | 0.000021 |
| Aug 02 | @ Cubs | 0.000011 |
| Aug 03 | @ Brewers | 0.000007 |
| Aug 04 | @ Brewers | 0.000003 |
| Aug 05 | @ Brewers | 0.000001 |
The single most likely game is now tomorrow's game; not surprisingly the Phillies have roughly a one-in-four chance of losing tonight and tomorrow afternoon and just getting this whole mess over with. There's almost a fifty-fifty shot of it coming during the Cardinals series (47.2%, to be exact); a 26.4% chance of it happening on the West Coast swing, which is when I originally thought it would happen; and an 0.6% chance of it happening it after their return from the West Coast.
By the way, current records for all teams are available at baseball-reference.com. The only team to have 10,000 wins so far are the Giants, with 10,150; the Cubs will most likely be next to reach that milestone, with 9,943. The Braves will be the next to cross the 10,000-loss line, but they're 320 short so it'll take a few years.
If you noticed that those are all National League teams, that's not a coincidence. The Phillies, Giants, Cubs, and Braves started play in 1883, 1883, 1876, and 1876 respectively; the eight original AL teams -- today's Orioles, Red Sox, White Sox, Indians, Tigers, Twins, Yankees, and A's -- all started in 1901, when that league was founded.)
The Phillies aren't the team with the lowest winning percentage, not by a long shot; they're .468 all-time. The Rangers, Rockies, and Padres are a bit worse at .467, .466, and .463; the Devil Rays are .398 all time. But they're all expansion teams, and expansion teams are historically bad the Phillies do have the worst winning percentage of the original sixteen teams. (The original NL is the Braves, Phillies, Cubs, Cardinals, Dodgers, Giants, Reds, and Pirates.) You can see online the standings of the the eight original AL teams and NL teams graphed since 1901. It won't surprise anyone to learn the Yankees are the best AL team in that time, and the Giants are only a bit less surprisingly the best NL team.
the 7/7/07 baseball coincidence
From Monday's New York Times, With a Big Day Ahead, Marketers Are Turning to Numerology:
With standards that loose, it's easy to manufacture coincidence. By my count, the following teams have seven letters in either their city or their nickname, or fourteen letters in the combination of the two:
AL East: Toronto Blue Jays, New York Yankees, Baltimore Orioles
AL Central: Cleveland Indians, Detroit Tigers, Minnesota Twins, Chicago White Sox
AL West: Los Angeles Angels (of Anaheim), Seattle Mariners, Oakland Athletics, Texas Rangers
NL East: New York Mets, Atlanta Braves, Florida Marlins
NL Central: Milwaukee Brewers, Chicago Cubs, St. Louis Cardinals, Pittsburgh Pirates, Houston Astros, Cincinnati Reds
NL West: San Diego Padres, Los Angeles Dodgers, Arizona Diamondbacks, Colorado Rockies
It's probably easier to list the teams that aren't includes: the Red Sox, Devil Rays, Royals, Phillies and Nationals. If you pick six teams at random, there are (25 choose 6) = 177100 ways that they'll be among the 25 with at least one seven-letter part in their name, out of 593775 total ways to pick six teams -- thus there's a 30% chance that this would happen, if you're lenient with the definition of "seven-letter team name". (For the truly pedantic among you, the schedule isn't balanced -- so certain teams are more likely to play each other than you'd expect by chance -- and also FOX tends to favor large media markets and winning teams for their Saturday afternoon coverage. This makes me happy because it means lots of Phillies and Red Sox games, which works against the coincidence; but it also means lots of games involving New York, Los Angeles, or Chicago teams, which works for it.)
Over the rest of the season, this same sort of coincidence happens again on July 7, July 28, August 4, August 18 (but there are only two games that day), and September 1, for five out of the eleven weeks that remain. (FOX makes it difficult to find their schedule for the first half of the season -- obviously nobody would care about that because it already happened.)
(Yes, the city of St. Louis is actually called "Saint Louis" -- but when was the last time you saw it written out?)
Oh, and today's games start at 3:55 PM Eastern. Could one of them end at 7:07?
The winner’s proposal is to be seen on computer-generated signs behind home plate during the three Major League Baseball games that Fox plans to broadcast on Saturday in different parts of the country: Atlanta Braves-San Diego Padres, Los Angeles Angels-New York Yankees and Minnesota Twins-Chicago White Sox.
Wait a minute. “Atlanta,” seven letters. “Angeles,” seven letters. “New York,” seven letters. “Yankees,” likewise. Ditto “Chicago.”
And “Minnesota Twins” and “San Diego Padres” are each 14 letters long, twice 7.
Spooky.
With standards that loose, it's easy to manufacture coincidence. By my count, the following teams have seven letters in either their city or their nickname, or fourteen letters in the combination of the two:
AL East: Toronto Blue Jays, New York Yankees, Baltimore Orioles
AL Central: Cleveland Indians, Detroit Tigers, Minnesota Twins, Chicago White Sox
AL West: Los Angeles Angels (of Anaheim), Seattle Mariners, Oakland Athletics, Texas Rangers
NL East: New York Mets, Atlanta Braves, Florida Marlins
NL Central: Milwaukee Brewers, Chicago Cubs, St. Louis Cardinals, Pittsburgh Pirates, Houston Astros, Cincinnati Reds
NL West: San Diego Padres, Los Angeles Dodgers, Arizona Diamondbacks, Colorado Rockies
It's probably easier to list the teams that aren't includes: the Red Sox, Devil Rays, Royals, Phillies and Nationals. If you pick six teams at random, there are (25 choose 6) = 177100 ways that they'll be among the 25 with at least one seven-letter part in their name, out of 593775 total ways to pick six teams -- thus there's a 30% chance that this would happen, if you're lenient with the definition of "seven-letter team name". (For the truly pedantic among you, the schedule isn't balanced -- so certain teams are more likely to play each other than you'd expect by chance -- and also FOX tends to favor large media markets and winning teams for their Saturday afternoon coverage. This makes me happy because it means lots of Phillies and Red Sox games, which works against the coincidence; but it also means lots of games involving New York, Los Angeles, or Chicago teams, which works for it.)
Over the rest of the season, this same sort of coincidence happens again on July 7, July 28, August 4, August 18 (but there are only two games that day), and September 1, for five out of the eleven weeks that remain. (FOX makes it difficult to find their schedule for the first half of the season -- obviously nobody would care about that because it already happened.)
(Yes, the city of St. Louis is actually called "Saint Louis" -- but when was the last time you saw it written out?)
Oh, and today's games start at 3:55 PM Eastern. Could one of them end at 7:07?
06 July 2007
liquor comes in fifths, but not tenths
I bought a fifth of vodka to bring to a party tonight. (Why, you ask? They said to bring something. And there often comes a moment towards the end of the evening where I'm thinking "damn, these people should have bought more vodka".) A fifth used to be a fifth of a gallon; now it's 750 mL. A fifth of a gallon is 756 mL, so the name makes sense. I'm kind of curious as to why this is a traditional unit, though. How Many?: A Dictionary of Units of Measurement (which is tremendously interesting, especially when you have other things you should be doing) says that it's an American version of a traditional British unit called the "bottle", which was one-sixth of an Imperial gallon, or 758 milliliters. The fact that an Imperial gallon is very nearly six-fifths of a U.S. gallon is apparently a coincidence.
While I was at the liquor store I noticed that they also sold vodka in 375 mL bottles. To my surprise, this is not called a "tenth". There are 28,800 google hits for "fifth of vodka" and 2 google hits for "tenth of vodka". The aforementioned unit dictionary mentions that a "half bottle of champagne" (375 mL) is called a "fillette".
Some people believe that a fifth and a liter are the same thing.
Another common size is the 1.75-liter bottle, often called a handle, and that size makes no sense at all; it's not two fifths, it's not a half-gallon, it's not all that round of a number in the metric system. I would have expected the "larger" size to be 1.5 liters (two fifths) or 2 liters. I thought it might be something to do with European law, but European law says that spirits must be sold in bottles of 0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.5, 1, 1.5, 2, 2.5, or 3 liters. Notice that neither .75 nor 1.75 appears here. From what I can gather, the reason for these laws is so that a company can't decrease the size of their bottle slightly and sell it for the same price, so price hikes are actually visible to the consumer. 0.75 liters is on the list of allowed bottle sizes for wine, though.
While I was at the liquor store I noticed that they also sold vodka in 375 mL bottles. To my surprise, this is not called a "tenth". There are 28,800 google hits for "fifth of vodka" and 2 google hits for "tenth of vodka". The aforementioned unit dictionary mentions that a "half bottle of champagne" (375 mL) is called a "fillette".
Some people believe that a fifth and a liter are the same thing.
Another common size is the 1.75-liter bottle, often called a handle, and that size makes no sense at all; it's not two fifths, it's not a half-gallon, it's not all that round of a number in the metric system. I would have expected the "larger" size to be 1.5 liters (two fifths) or 2 liters. I thought it might be something to do with European law, but European law says that spirits must be sold in bottles of 0.02, 0.03, 0.04, 0.05, 0.10, 0.20, 0.5, 1, 1.5, 2, 2.5, or 3 liters. Notice that neither .75 nor 1.75 appears here. From what I can gather, the reason for these laws is so that a company can't decrease the size of their bottle slightly and sell it for the same price, so price hikes are actually visible to the consumer. 0.75 liters is on the list of allowed bottle sizes for wine, though.
the density of money
I have a jar of assorted coins. I sort out the quarters separately (because I need them for laundry) but otherwise every so often I throw my change in here. It's getting kind of heavy; one of these days I'll cash it in for real money.
As I was lifting it this morning, I began to wonder -- if I knew how much it weighed, could I tell from that approximately how much money it contained?
Then I went to buy breakfast -- which cost me $5.25, and I paid with a $20 bill. I got $14.75 in change -- a ten, four ones, seven dimes and a nickel, because there were no quarters. The woman working the cash register said she was sorry they were out of quarters; I replied that dimes were okay because at least they're small.
So, here's the question: what's the density of money in, say, dollars per kilogram?
A U.S. dime weighs 2.268 grams; that's $44.09 per kilogram.
A U.S. quarter weighs 5.670 grams; that's also $44.09 per kilogram.
A U. S. nickel weighs 5.000 grams; that's $10.00 per kilogram. Wikipedia says that"nickels have always had a value of one cent per gram", which is interesting if true in part because nickels have been minted since 1866. Clearly the designers of the nickel were thinking in metric.
A U.S. penny weighs 2.5 grams; that's $4.00 per kilogram.
I presume the fact that the quarter is exactly two and one half times the weight of the dime has something to do with the fact that they're both made out of the same alloy, an 11:1 copper/nickel mixture; the nickel is three-fourths copper and one-fourth nickel; the penny is 97.5% zinc and 2.5% copper.
Right now, metalprices.com reports that copper sells for $7.895/kilogram; zinc, $3.436/kilogram; nickel, $36.20/kilogram. Thus dimes and quarters, if you melt them down, could sell for $10.25/kg; nickels, $14.97/kg; pennies, $3.54/kg. I'm surprised to learn that nickels are worth less than the metal underlying them, because you hear this more often about pennies, even though it's not actually true. It does, however, cost more to make a penny than that coin is worth, and the U. S. Mint has passed regulations about the melting down of pennies and nickels.
But what's the density of "money"? That's a bit trickier. Let's assume that on any given transaction, I pay with a whole number of dollars; furthermore assume that the "fractional part" of my change is equally likely to be 0, 1, 2, ..., 99 cents, and that it's given back to me with the smallest number of coins possible. The easiest way to do the computation is to assume that I make 100 transactions, in which I get 0, 1, 2, ..., 99 cents back. Now I have $49.50. How much does it weigh?
Well, twenty times I got 0 pennies; twenty times I got 1 penny; and so on up to 4 pennies. So I have 20*(0+1+2+3+4) = 200 pennies.
I'll never get more than one nickel. I get a nickel if the fractional part of my change is 5-9, 15-19, 30-34, 40-44, 55-59, 65-69, 80-84, or 90-94 cents; there are 40 numbers there. So I get 40 nickels.
The rest of what I get is dimes and quarters; since dimes and quarters have the same "money density" I won't distinguish between them. I get $45.50 worth of dimes and quarters. (In fact, I get 150 quarters and 80 dimes.)
Together, all these coins weigh 1732 grams; thus the density of money appears to be $28.58 per kilogram.
But in reality, it won't be nearly this much. I try to get rid of change when I'm carrying it, and a lot of businesses now set their prices so that they don't have to deal with nickels. (At one of my favorite coffee shops, all the prices are multiples of 25 cents. The problem with this is that people will bitch and moan when that inevitable day comes when they raise the price of a large coffee from $1.75 to $2; if they were willing to deal with nickels they'd only have to raise it to $1.80.) And I'm more likely to spend quarters than any other coin, because they work the laundry machine (They buy newspapers, too; the Inquirer costs 50 cents. The machines take nickels, dimes, or quarters, but usually I use two quarters.)
I can't weigh my jar of money to tell you what its actual density is -- I don't have a scale. But when I cash it in I'll let you know how many of each kind of coin the coin-counting machine says it had.
As I was lifting it this morning, I began to wonder -- if I knew how much it weighed, could I tell from that approximately how much money it contained?
Then I went to buy breakfast -- which cost me $5.25, and I paid with a $20 bill. I got $14.75 in change -- a ten, four ones, seven dimes and a nickel, because there were no quarters. The woman working the cash register said she was sorry they were out of quarters; I replied that dimes were okay because at least they're small.
So, here's the question: what's the density of money in, say, dollars per kilogram?
A U.S. dime weighs 2.268 grams; that's $44.09 per kilogram.
A U.S. quarter weighs 5.670 grams; that's also $44.09 per kilogram.
A U. S. nickel weighs 5.000 grams; that's $10.00 per kilogram. Wikipedia says that"nickels have always had a value of one cent per gram", which is interesting if true in part because nickels have been minted since 1866. Clearly the designers of the nickel were thinking in metric.
A U.S. penny weighs 2.5 grams; that's $4.00 per kilogram.
I presume the fact that the quarter is exactly two and one half times the weight of the dime has something to do with the fact that they're both made out of the same alloy, an 11:1 copper/nickel mixture; the nickel is three-fourths copper and one-fourth nickel; the penny is 97.5% zinc and 2.5% copper.
Right now, metalprices.com reports that copper sells for $7.895/kilogram; zinc, $3.436/kilogram; nickel, $36.20/kilogram. Thus dimes and quarters, if you melt them down, could sell for $10.25/kg; nickels, $14.97/kg; pennies, $3.54/kg. I'm surprised to learn that nickels are worth less than the metal underlying them, because you hear this more often about pennies, even though it's not actually true. It does, however, cost more to make a penny than that coin is worth, and the U. S. Mint has passed regulations about the melting down of pennies and nickels.
But what's the density of "money"? That's a bit trickier. Let's assume that on any given transaction, I pay with a whole number of dollars; furthermore assume that the "fractional part" of my change is equally likely to be 0, 1, 2, ..., 99 cents, and that it's given back to me with the smallest number of coins possible. The easiest way to do the computation is to assume that I make 100 transactions, in which I get 0, 1, 2, ..., 99 cents back. Now I have $49.50. How much does it weigh?
Well, twenty times I got 0 pennies; twenty times I got 1 penny; and so on up to 4 pennies. So I have 20*(0+1+2+3+4) = 200 pennies.
I'll never get more than one nickel. I get a nickel if the fractional part of my change is 5-9, 15-19, 30-34, 40-44, 55-59, 65-69, 80-84, or 90-94 cents; there are 40 numbers there. So I get 40 nickels.
The rest of what I get is dimes and quarters; since dimes and quarters have the same "money density" I won't distinguish between them. I get $45.50 worth of dimes and quarters. (In fact, I get 150 quarters and 80 dimes.)
Together, all these coins weigh 1732 grams; thus the density of money appears to be $28.58 per kilogram.
But in reality, it won't be nearly this much. I try to get rid of change when I'm carrying it, and a lot of businesses now set their prices so that they don't have to deal with nickels. (At one of my favorite coffee shops, all the prices are multiples of 25 cents. The problem with this is that people will bitch and moan when that inevitable day comes when they raise the price of a large coffee from $1.75 to $2; if they were willing to deal with nickels they'd only have to raise it to $1.80.) And I'm more likely to spend quarters than any other coin, because they work the laundry machine (They buy newspapers, too; the Inquirer costs 50 cents. The machines take nickels, dimes, or quarters, but usually I use two quarters.)
I can't weigh my jar of money to tell you what its actual density is -- I don't have a scale. But when I cash it in I'll let you know how many of each kind of coin the coin-counting machine says it had.
05 July 2007
changing horses (i. e. managers) in midstream (midseason)
Over at Mike's Baseball Rants, Mike Carminati has analyzed what happens to baseball teams when they change managers in midseason, and he wonders: how does a team fare when it changes its manager?
What got my attention was the statement that on average, if a team changes its manager its winning percentage improves by .044 in the next season.
After all, we ought to expect some sort of regression to the mean. The average team which changes its manager is a .445 team. (Over 162 games, that's 72 wins and 90 losses.) I figure that the fact that the team wins less than half its game is some combination of lack of skill and bad luck; one expects that they'll do better next year. A team which is "naturally" a .500 team should do this poorly one year out of every fourteen or so.
And indeed, if we compare the winning percentage of a team in year N to the same team in year N+1, the winning percentage in year N+1 is given by, roughly,
PCTn+1 = 0.8 * PCTn + 0.1
I'm being deliberate in only giving one significant figure here; depending on exactly which sample I use the coefficients change quite a bit. But no matter what happens, it seems like a team's record regresses one-fifth of the way back to the mean. A 71-91 team this year (ten games below .500) ought to be eight below 500, or 73-89, next year; a 111-51 team this year (thirty games above .500) ought to be 105-57 next year.
So you'd expect that .445 team this year to improve to .456 next year -- an improvement of .011. There is therefore some real effect from changing the manager.
However, midseason replacements don't seem to have much of an effect. Carminati also computes that when a team changes managers in midseason, they do .007 better after the replacement than before. It would be too much effort to compile how teams do on average in the first and second half of the season, but I'd expect some sort of regression to the mean. It's not immediately clear how this should compare with the regression to the mean between consecutive seasons. On the one hand, the samples are smaller, so you'd expect a bit more volatility -- but does that mean that our hypothetical .445 team over the first half should do better or worse than .456 in the second half? On the other hand, there's a lot more turnover during the offseason just because there's more time there; I'd definitely expect the regression to be stronger between seasons than between halves of the same season, because a team which has done badly will shop around and get itself some talent, and a team which has done well is likely to be paying out too much and will sell off some talent. So this .007 figure might not even be significant. I suspect that a lot matters on the particular situation that the team is in.
What got my attention was the statement that on average, if a team changes its manager its winning percentage improves by .044 in the next season.
After all, we ought to expect some sort of regression to the mean. The average team which changes its manager is a .445 team. (Over 162 games, that's 72 wins and 90 losses.) I figure that the fact that the team wins less than half its game is some combination of lack of skill and bad luck; one expects that they'll do better next year. A team which is "naturally" a .500 team should do this poorly one year out of every fourteen or so.
And indeed, if we compare the winning percentage of a team in year N to the same team in year N+1, the winning percentage in year N+1 is given by, roughly,
PCTn+1 = 0.8 * PCTn + 0.1
I'm being deliberate in only giving one significant figure here; depending on exactly which sample I use the coefficients change quite a bit. But no matter what happens, it seems like a team's record regresses one-fifth of the way back to the mean. A 71-91 team this year (ten games below .500) ought to be eight below 500, or 73-89, next year; a 111-51 team this year (thirty games above .500) ought to be 105-57 next year.
So you'd expect that .445 team this year to improve to .456 next year -- an improvement of .011. There is therefore some real effect from changing the manager.
However, midseason replacements don't seem to have much of an effect. Carminati also computes that when a team changes managers in midseason, they do .007 better after the replacement than before. It would be too much effort to compile how teams do on average in the first and second half of the season, but I'd expect some sort of regression to the mean. It's not immediately clear how this should compare with the regression to the mean between consecutive seasons. On the one hand, the samples are smaller, so you'd expect a bit more volatility -- but does that mean that our hypothetical .445 team over the first half should do better or worse than .456 in the second half? On the other hand, there's a lot more turnover during the offseason just because there's more time there; I'd definitely expect the regression to be stronger between seasons than between halves of the same season, because a team which has done badly will shop around and get itself some talent, and a team which has done well is likely to be paying out too much and will sell off some talent. So this .007 figure might not even be significant. I suspect that a lot matters on the particular situation that the team is in.
swarm theory
National Geographic on Swarm Theory -- the field variously known as "complex systems", "emergent behavior", etc. In some sense certain complex systems are greater than the sum of their parts -- an ant colony seems to have some sort of "intelligence", even though a single ant doesn't. Each individual person betting on a horse race has their biases, but in parimutuel betting the final odds (which depend on how many people bet on each horse) very nearly reflect what actually happens. Juries are smarter than each of their individual members -- although only when the jurors act independently. (This makes me wonder: are people better or worse at following complex arguments than they were a couple hundred years ago?)
Public transportation systems, especially in the U.S., are notoriously inefficient; I suspect a large part of this is because they run on the same routes they've historically run on. I have to wonder if this sort of thing could be applied to redesign of public transit; the article mentions other logistical applications. Southwest Airlines, for example, has used this for scheduling. And American Air Liquide, which delivers gases made at a large number of plants to a large number of customers, has redone a lot of their delivery this way.
Unfortunately, though, Air Liquide has found that some of their drivers have to do unintuitive things -- they're no longer delivering from the plant nearest the customer to the customer. I have a feeling this wouldn't work well with people; you can't tell someone "well, it's going to take you an extra half-hour to get to work today, but that saves fifty other people one minute!" even though it does reduce total transit time.
I suspect there might be applications to, say, the dispatching of cabs, or the placing of shared cars (disclaimer: I am not a member of Philly Car Share but lots of people I know are). Or to a hypothetical company like the one Scott Adams suggested which would get people rides by having them text-message their destination to some central server and then hook them up with someone who's willing to give them a ride. Although the appropriate metaphor in this situation is actually more like traditional, decentralized hitchhiking. Hey, maybe that's the solution to the energy crisis!
Public transportation systems, especially in the U.S., are notoriously inefficient; I suspect a large part of this is because they run on the same routes they've historically run on. I have to wonder if this sort of thing could be applied to redesign of public transit; the article mentions other logistical applications. Southwest Airlines, for example, has used this for scheduling. And American Air Liquide, which delivers gases made at a large number of plants to a large number of customers, has redone a lot of their delivery this way.
Unfortunately, though, Air Liquide has found that some of their drivers have to do unintuitive things -- they're no longer delivering from the plant nearest the customer to the customer. I have a feeling this wouldn't work well with people; you can't tell someone "well, it's going to take you an extra half-hour to get to work today, but that saves fifty other people one minute!" even though it does reduce total transit time.
I suspect there might be applications to, say, the dispatching of cabs, or the placing of shared cars (disclaimer: I am not a member of Philly Car Share but lots of people I know are). Or to a hypothetical company like the one Scott Adams suggested which would get people rides by having them text-message their destination to some central server and then hook them up with someone who's willing to give them a ride. Although the appropriate metaphor in this situation is actually more like traditional, decentralized hitchhiking. Hey, maybe that's the solution to the energy crisis!
04 July 2007
what good is an "average"?
Ugly Airline Math: Planes Late, Fliers Even Later -- from the July 5 New York Times.
Basically, airlines count the "average delay" by how late the average plane is. But what really matters is how late the average passenger is, which is a different problem because of connecting flights. If you're scheduled to have a ninety-minute layover and your incoming flight is forty minutes late, then suddenly your connection is tighter but you'll be okay, and moreover you get to your destination at the same time you would have otherwise. But if your incoming flight is two hours late, then you have to get on the next flight -- which means you'll be far more than two hours delayed.
The average delay for planes is 15 minutes; for passengers, 25.
But is this statistic even meaningful? The article talks of people who were, say, 12 or 24 hours late to their destinations. The mean isn't meaningful for these sorts of distributions, which are very skewed. Sure, it's easy to calculate. But if I were a passenger, and I had to pick one number to know about an airline's performance, it wouldn't be the mean delay. I'd want to know what percentage of their passengers get where they're going more than, say, an hour late. Or which percentage get there not on the flight they were originally ticketed on. (The same flight number on the next day doesn't count as the "same flight", although I bet they'd find a way to make it count.)
But whatever you do, the airlines will probably find a way to game the system so that they look good under the performance metric in question without actually satisfying their customers. This is because people will demand some simple metric they can understand, even though I suspect that any single number that measures an airline's performance would be something like, say, Google's PageRank -- impossible for the layperson to calculate but still remarkably useful.
Basically, airlines count the "average delay" by how late the average plane is. But what really matters is how late the average passenger is, which is a different problem because of connecting flights. If you're scheduled to have a ninety-minute layover and your incoming flight is forty minutes late, then suddenly your connection is tighter but you'll be okay, and moreover you get to your destination at the same time you would have otherwise. But if your incoming flight is two hours late, then you have to get on the next flight -- which means you'll be far more than two hours delayed.
The average delay for planes is 15 minutes; for passengers, 25.
But is this statistic even meaningful? The article talks of people who were, say, 12 or 24 hours late to their destinations. The mean isn't meaningful for these sorts of distributions, which are very skewed. Sure, it's easy to calculate. But if I were a passenger, and I had to pick one number to know about an airline's performance, it wouldn't be the mean delay. I'd want to know what percentage of their passengers get where they're going more than, say, an hour late. Or which percentage get there not on the flight they were originally ticketed on. (The same flight number on the next day doesn't count as the "same flight", although I bet they'd find a way to make it count.)
But whatever you do, the airlines will probably find a way to game the system so that they look good under the performance metric in question without actually satisfying their customers. This is because people will demand some simple metric they can understand, even though I suspect that any single number that measures an airline's performance would be something like, say, Google's PageRank -- impossible for the layperson to calculate but still remarkably useful.
how to properly celebrate this day
To properly celebrate this day, you should have eleven drinks.
Why eleven? Because the United States turns 231 years old today. Twenty-one is the legal drinking age. 231 is eleven times twenty-one.
I'd do it, but I don't hold my liquor that well.
And just think! In 2028, when the U.S. turns 252, you should drink twelve times. And vote fourteen times. Except there will most likely be no election to vote in on that particular day, and voting more than once -- at least at the present writing -- is illegal. May I suggest having sex fourteen times instead, if you live in a state where the age of consent is 18?. If you live in South Carolina, and are female, you should celebrate that day by having sex eighteen times, since the age of consent for females there is fourteen. You could also die for your country fourteen times, but that would hurt, and it would be impossible. Unless you are one and two-thirds cats.
Why eleven? Because the United States turns 231 years old today. Twenty-one is the legal drinking age. 231 is eleven times twenty-one.
I'd do it, but I don't hold my liquor that well.
And just think! In 2028, when the U.S. turns 252, you should drink twelve times. And vote fourteen times. Except there will most likely be no election to vote in on that particular day, and voting more than once -- at least at the present writing -- is illegal. May I suggest having sex fourteen times instead, if you live in a state where the age of consent is 18?. If you live in South Carolina, and are female, you should celebrate that day by having sex eighteen times, since the age of consent for females there is fourteen. You could also die for your country fourteen times, but that would hurt, and it would be impossible. Unless you are one and two-thirds cats.
the first and the fourth of July
It's often been pointed out by Americans that for some reason, a lot of countries have their national holidays in July.
This is because if Americans can remember the national holidays of two countries, they're the U.S. (July 4) and Canada (July 1); if they can remember a third there's a good chance it's France (July 14).
What are the chances that two countries which border each other have their national days within three days of each other? You can pick when the first one should be at random, on the Nth day of the year; then the other one must have its day sometime between N-3 and N+3, a seven-day span So the probability is one in fifty-two; one expects there to be two countries which border each other and have national days within three days of each other.
Two bordering countries have a one-in-365 chance of having the same national day, if such days are chosen at random. But since such days often commemorate historical events, and two adjacent countries probably share some history, I'd think that such events are more likely than one in 365.
Of course, this is all made a bit trickier by the fact that some countries seem to have more than one "national day". Mexico's Independence Day, for example, is September 16 -- but I bet a lot of Americans thinks it's May 5. Canada celebrates July 1 -- but Quebec calls June 24 "la fête nationale du Québec".
A quick look at the Wikipedia lisf of national holiays reveals the following coincidences:
The bunch of Central American countries on September 15 all commemorate the same event, the formation of the Federal Republic of Central America in 1821, which split up into those countries about twenty years later. Mexico declared its independence on September 16, 1810.
The Canadian and American holidays of course celebrate different events -- unless somehow Canada started in Philadelphia and nobody told me. (Incidentally, the Continental Congress actually voted for independence on July 2. On this basis I think I should not have had to serve jury duty on Monday, since that should have been a holiday -- and less than a mile from where it all happened, no less!
The Pakistan and India days seem to both commemorate the 1947 partition of India; it's not clear to me why they're not the same day.
But there seem to be at least two cases in which adjacent countries celebrate their national days within three days of each other and they're not commemorating the same event -- namely Mexico-Guatemala and U.S.-Canada. (And, of course, the U.S. borders Mexico.) This is less than I would have expected -- you'd expect two such near-collisions if there were about 104 borders between countries in the world, when there are clearly more than that. But I am working from a list which is clearly incomplete. This list is more complete but not sorted by date.
This is because if Americans can remember the national holidays of two countries, they're the U.S. (July 4) and Canada (July 1); if they can remember a third there's a good chance it's France (July 14).
What are the chances that two countries which border each other have their national days within three days of each other? You can pick when the first one should be at random, on the Nth day of the year; then the other one must have its day sometime between N-3 and N+3, a seven-day span So the probability is one in fifty-two; one expects there to be two countries which border each other and have national days within three days of each other.
Two bordering countries have a one-in-365 chance of having the same national day, if such days are chosen at random. But since such days often commemorate historical events, and two adjacent countries probably share some history, I'd think that such events are more likely than one in 365.
Of course, this is all made a bit trickier by the fact that some countries seem to have more than one "national day". Mexico's Independence Day, for example, is September 16 -- but I bet a lot of Americans thinks it's May 5. Canada celebrates July 1 -- but Quebec calls June 24 "la fête nationale du Québec".
A quick look at the Wikipedia lisf of national holiays reveals the following coincidences:
- Canada has Canada Day on July 1; the U. S. has Independence Day on July 4
- Pakistan has Independence Day on August 14; India has Independence Day on August 15
- Costa Rica, El Salvador, Guatemala, Honduras, and Nicaragaua all have Independence Day on September 15; Mexico has it on September 16. (Chile declared its independence from Spain two days after Mexico did, but it looks like they were separate events. Belize declared Independence on September 21, but over a century later.)
The bunch of Central American countries on September 15 all commemorate the same event, the formation of the Federal Republic of Central America in 1821, which split up into those countries about twenty years later. Mexico declared its independence on September 16, 1810.
The Canadian and American holidays of course celebrate different events -- unless somehow Canada started in Philadelphia and nobody told me. (Incidentally, the Continental Congress actually voted for independence on July 2. On this basis I think I should not have had to serve jury duty on Monday, since that should have been a holiday -- and less than a mile from where it all happened, no less!
The Pakistan and India days seem to both commemorate the 1947 partition of India; it's not clear to me why they're not the same day.
But there seem to be at least two cases in which adjacent countries celebrate their national days within three days of each other and they're not commemorating the same event -- namely Mexico-Guatemala and U.S.-Canada. (And, of course, the U.S. borders Mexico.) This is less than I would have expected -- you'd expect two such near-collisions if there were about 104 borders between countries in the world, when there are clearly more than that. But I am working from a list which is clearly incomplete. This list is more complete but not sorted by date.
03 July 2007
10,000 losses: an update
A week ago, I posted a forecast of the date of the 10,000th Phillies loss. In particular I wondered what the chances were that I'd witness this historic event. I have tickets for July 13.
Since then, the Phillies have won two and lost five (including dropping three out of four to the Mets, which was particularly galling because New York fans have started coming down to Philly in large number for the games). At the time that I wrote that previous post, the Phillies had 9,991 losses; now they have 9,996, so there are four more to go.
Fortunately, the 10,000th loss can't come tomorrow, on July 4; even if they lose tonight's game (which is just getting underway) and tomorrow's, that'll "only" be 9,998. The earliest the 10,000th loss could come is against the Rockies on Saturday.
But here are the probabilities now:
In particular, the peak now looks like the Cardinals series; there's a 35% chance of it happening in those three days, and nearly a one-in-eight chance I'll witness the historic event in person. My original prediction had a 66.8% chance of it happening on the West Coast swing July 16-22; now it's only 46.3%. The effect is still helped by the fact that the Dodgers and Padres are strong teams; notice that the probability decreases from the 14th to the 15th and then increases from the 15th to the 16th, and that the 10,000th loss is three times as likely to come on the 22nd as on the 24th. And the distribution doesn't stretch nearly as far into the future; the first game which has a chance of less than one in two million to be the 10000th loss (which rounds to zero) is August 14 in the current simulation, versus August 27 when I ran the numbers last week.
In other baseball-milestone news, Clay Davenport of Baseball Prospectus made a prediction in May that Barry Bonds was most likely to hit his 756th home run in mid-June, with a probability of 80% that he'd have done it by now. He's up to 751. This is because he got off to a slow start. If I had to guess, I'd predict as follows: Bonds has hit 17 home runs in his team's first 81 games. (The Giants are currently playing their 81st game, and are in the fourth innings; Bonds hit a home run in the first. I'm assuming he doesn't hit any more tonight.) So it'll take him 81 * 5/17 = 24 more games to reach the record, which projects to July 31 against the Dodgers. This sort of logic is notoriously bad; it's the sort of logic that says that since a player hits 12 home runs in April he'll hit 72 for the season, or that a team that starts its season by winning three out of four will go on to have a 122-40 record, when of course there's really regression to the mean. But it seems at least somewhat sound in this case.
Since then, the Phillies have won two and lost five (including dropping three out of four to the Mets, which was particularly galling because New York fans have started coming down to Philly in large number for the games). At the time that I wrote that previous post, the Phillies had 9,991 losses; now they have 9,996, so there are four more to go.
Fortunately, the 10,000th loss can't come tomorrow, on July 4; even if they lose tonight's game (which is just getting underway) and tomorrow's, that'll "only" be 9,998. The earliest the 10,000th loss could come is against the Rockies on Saturday.
But here are the probabilities now:
| Jul 03 | @ Astros | 0.000000 |
| Jul 04 | @ Astros | 0.000000 |
| Jul 06 | @ Rockies | 0.000000 |
| Jul 07 | @ Rockies | 0.047428 |
| Jul 08 | @ Rockies | 0.110681 |
| Jul 13 | v. Cardinals | 0.117107 |
| Jul 14 | v. Cardinals | 0.121772 |
| Jul 15 | v. Cardinals | 0.115521 |
| Jul 16 | @ Dodgers | 0.152669 |
| Jul 17 | @ Dodgers | 0.118578 |
| Jul 18 | @ Dodgers | 0.083456 |
| Jul 19 | @ Padres | 0.054027 |
| Jul 20 | @ Padres | 0.033566 |
| Jul 21 | @ Padres | 0.019969 |
| Jul 22 | @ Padres | 0.011472 |
| Jul 24 | v. Nationals | 0.003890 |
| Jul 25 | v. Nationals | 0.002815 |
| Jul 26 | v. Nationals | 0.002028 |
| Jul 27 | v. Pirates | 0.001477 |
| Jul 28 | v. Pirates | 0.001050 |
| Jul 29 | v. Pirates | 0.000744 |
| Jul 30 | @ Cubs | 0.000736 |
| Jul 31 | @ Cubs | 0.000431 |
| Aug 01 | @ Cubs | 0.000250 |
| Aug 02 | @ Cubs | 0.000144 |
| Aug 03 | @ Brewers | 0.000099 |
| Aug 04 | @ Brewers | 0.000048 |
| Aug 05 | @ Brewers | 0.000023 |
| Aug 07 | v. Marlins | 0.000007 |
| Aug 08 | v. Marlins | 0.000005 |
| Aug 09 | v. Marlins | 0.000003 |
| Aug 10 | v. Braves | 0.000002 |
| Aug 11 | v. Braves | 0.000001 |
| Aug 12 | v. Braves | 0.000001 |
| Aug 14 | @ Nationals | 0.000000 |
In particular, the peak now looks like the Cardinals series; there's a 35% chance of it happening in those three days, and nearly a one-in-eight chance I'll witness the historic event in person. My original prediction had a 66.8% chance of it happening on the West Coast swing July 16-22; now it's only 46.3%. The effect is still helped by the fact that the Dodgers and Padres are strong teams; notice that the probability decreases from the 14th to the 15th and then increases from the 15th to the 16th, and that the 10,000th loss is three times as likely to come on the 22nd as on the 24th. And the distribution doesn't stretch nearly as far into the future; the first game which has a chance of less than one in two million to be the 10000th loss (which rounds to zero) is August 14 in the current simulation, versus August 27 when I ran the numbers last week.
In other baseball-milestone news, Clay Davenport of Baseball Prospectus made a prediction in May that Barry Bonds was most likely to hit his 756th home run in mid-June, with a probability of 80% that he'd have done it by now. He's up to 751. This is because he got off to a slow start. If I had to guess, I'd predict as follows: Bonds has hit 17 home runs in his team's first 81 games. (The Giants are currently playing their 81st game, and are in the fourth innings; Bonds hit a home run in the first. I'm assuming he doesn't hit any more tonight.) So it'll take him 81 * 5/17 = 24 more games to reach the record, which projects to July 31 against the Dodgers. This sort of logic is notoriously bad; it's the sort of logic that says that since a player hits 12 home runs in April he'll hit 72 for the season, or that a team that starts its season by winning three out of four will go on to have a 122-40 record, when of course there's really regression to the mean. But it seems at least somewhat sound in this case.
Five dollars a year for energy independence?
In today's Philadelphia Inquirer: A tax, or any other name, would raise ire.
Pennsylvania governor Ed Rendell is attempting to create an "Energy Independence Fund" by raising the cost of electricity by $0.0005 per kilowatt-hour. The average price of electricity in Pennsylvania is between $0.09 and $0.11 per kilowatt-hour, depending on who you believe. Rendell claims that this would cost the average household "the equivalent of half a cup of coffee per month"; more precisely it's estimated that the cost of this program would be $5.40 per year for the average residential customer. (That works out to 900 kilowatt-hours per household, which is at least in the right ballpark.) This would create a fund of $850 million which would supposedly save Pennsylvanians about $1 billion per year, by encouraging the use of alternative fuels (for example, Pennsylvania has no oil, but we can grow corn for ethanol!), creating rebate programs for people who buy more efficient appliances or install solar panels, and so on.
House minority leader Sam Smith says:
What Smith is saying, it seems, is that some people use more electricity than average. But any reasonable definition of "average" has this property! Besides, we're talking about raising electricity bills by half a percent here. Al Gore supposedly uses 221,000 kilowatt-hours per year; even his bill would only go up by $9 a month or so. Besides, if this encourages people to use half a percent less electricity, it's a good idea. And in the end, people's bills will be reduced assuming these programs work out.
And I'd guess that the national average of 10,656 kilowatt-hours per year per household (as cited in the article re: Gore, and implied by the $5.40/year and $0.0005/kW-h figures) is mean use, not median use. The distribution of electricity use is probably skewed to the right, so the median is lower than the mean. (A lot of distributions which occur in economics are like this; one of the standard examples these days are real estate prices. Since there are a few really expensive houses out there, it's felt that the median is a better indicator of the market as a whole than the mean is, so the median is what's cited when people talk about whether real estate prices are rising or falling.) So you could even make the argument that Rendell is overstating the impact of the new surcharge.
In case you're wondering, I figure that this tax would cost me, directly, less than $1 a year. (Indirectly it'll cost a bit more, because the people I buy things from will also see their electricity prices raised.) I could have made more than $1 in the time it took me to read the article and write this post.
Pennsylvania governor Ed Rendell is attempting to create an "Energy Independence Fund" by raising the cost of electricity by $0.0005 per kilowatt-hour. The average price of electricity in Pennsylvania is between $0.09 and $0.11 per kilowatt-hour, depending on who you believe. Rendell claims that this would cost the average household "the equivalent of half a cup of coffee per month"; more precisely it's estimated that the cost of this program would be $5.40 per year for the average residential customer. (That works out to 900 kilowatt-hours per household, which is at least in the right ballpark.) This would create a fund of $850 million which would supposedly save Pennsylvanians about $1 billion per year, by encouraging the use of alternative fuels (for example, Pennsylvania has no oil, but we can grow corn for ethanol!), creating rebate programs for people who buy more efficient appliances or install solar panels, and so on.
House minority leader Sam Smith says:
I would venture a guess that any household of three or four with a few extra appliances or maybe air-conditioning will pay more," he said. "It's a big tax on Pennsylvanians, and he is using an average number to downplay a bigger tax than it is."
What Smith is saying, it seems, is that some people use more electricity than average. But any reasonable definition of "average" has this property! Besides, we're talking about raising electricity bills by half a percent here. Al Gore supposedly uses 221,000 kilowatt-hours per year; even his bill would only go up by $9 a month or so. Besides, if this encourages people to use half a percent less electricity, it's a good idea. And in the end, people's bills will be reduced assuming these programs work out.
And I'd guess that the national average of 10,656 kilowatt-hours per year per household (as cited in the article re: Gore, and implied by the $5.40/year and $0.0005/kW-h figures) is mean use, not median use. The distribution of electricity use is probably skewed to the right, so the median is lower than the mean. (A lot of distributions which occur in economics are like this; one of the standard examples these days are real estate prices. Since there are a few really expensive houses out there, it's felt that the median is a better indicator of the market as a whole than the mean is, so the median is what's cited when people talk about whether real estate prices are rising or falling.) So you could even make the argument that Rendell is overstating the impact of the new surcharge.
In case you're wondering, I figure that this tax would cost me, directly, less than $1 a year. (Indirectly it'll cost a bit more, because the people I buy things from will also see their electricity prices raised.) I could have made more than $1 in the time it took me to read the article and write this post.
02 July 2007
jury duty: coincidences, and semi-juries
Today I had jury duty.
During the lunch break I went to Reading Terminal Market, where I spent $9.05 for lunch (it was a large lunch, because I didn't want to be sitting around hungry); my pay for one day of jury duty was $9. (I was not chosen for a trial.) Coincidence? Probably, because I wasn't thinking "I'm spending my nine bucks on lunch" when I was walking around choosing where I would buy it.
Then I wandered down to a bookstore and found myself flipping through Steven Landsburg's book More Sex Is Safer Sex: The Unconventional Wisdom of Economics
. . (I didn't buy it; it seemed interesting, but not $26 worth of interesting.) In particular, this book suggests that the jury system is broken (the link goes to the Freakonomics blogs, where he was interviewed a few weeks ago). The basic idea is that jurors have no incentive to do a good job. This is clearly true, although I'm not sure how to incentivize the jury system. When I got home I ran across an entry in the Freakonomics blog which mentioned that book. Coincidence? Maybe. Maybe not. (And I thin I saw the original Landsburg interview a few weeks ago and forgot about it, which may have primed me to be more likely to look at that specific book.)
Yet another coincidence: I went to high school with the judge's son. (I don't think this is how I got out of serving.) I also went to high school with the son of one of my panel-mates. It occurred to me as I was heading in this morning that if a few hundred people were called today, the chances I'd know one of them were not bad; there are probably about a million adults in Philadelphia, of whom I know a few hundred. I don't think anyone I know was there today (if so, I didn't see them) but as I said there were parents of people I knew. In some ways Philly is the largest small town in the country.
While waiting to be selected, it occurred to me that the voir dire procedure is set up so that no individual juror who is selected was biased. We were a panel of 50 for a sexual assault case, from which fourteen jurors were essentially selected; although I wasn't counting, I would guess that there were no more than twenty people who satisfied the following three conditions:
Still, would it have been such a horrible thing to have a sexual assault victim on the jury? A randomly selected panel of fourteen would probably have had at least one. I don't see why each individual juror has to be unbiased in order for the group as a whole to be unbiased.
Finally, some math. In Landsburg's book he suggests the following: break each jury up into two half-juries of six. If they come to the same conclusion, that's the verdict; he wasn't clear on what to do if they came to opposing conclusions. (Presumably it would be treated like current hung juries are.) In this study by Bruce Spencer it's suggested that juries are "right" about 88% of the time. This got me thinking -- how likely does this mean an individual juror is to be "right" about the verdict? If we assume that jurors make their decisions independently, that majority rules (which is a bit ingenuous because juries in criminal cases have to be unanimous), and throw out 6-6 results, it turns out each individual juror has to come to the right decision with probability 63.6% to recover this 88% probability. This is related to the post I made a couple weeks ago about the World Series; if one team is slightly better than another, they have a decent chance of winning a single game but not so good a chance of winning a whole series. The teams here are, of course, "guilty" and "not guilty".
So what does this say for the half-jury suggestion? Let's say that each juror, indepedently, has a 63.6% chance of being right, and there's a jury of six. Say the defendant is guilty. Then the probability that all six jurors will say this is (.636)6 = .066; the probability that five think he's guilty and one thinks he's innocent is 6(.636)5(.364) = .227; the probability of the 4-2 split is 15(.636)4(.364)2 = .325. So the probability of one of these three results is 0.618; the probability of having a 3-3 split is 20(.636)3(.364)3 = 0.248. So the probability of a half-jury finding the defendant guilty is (.618)/(1-.248) = .821. Not surprisingly, this is less than the chance of a full jury finding the defendant guilty.
But the chance that both half-juries find the defendant guilty is (.821)2 = .674; the chance that they both find him innocent (even though he did it!) is (1-.821)2 = .032. So the probability of finding the defendant guilty, given that there's a verdict at all, is .674/(.674+.032) = .955. In the end, this plan achieves much greater accuracy at the expense of increasing the number of hung juries. It seems worth considering, though. (Incidentally, you can't beat the hung jury problem by changing the sub-jury size; either at least one sub-jury is of even size or there's an even number of sub-juries, since 12 is even.)
I suspect, though, that this sort of thing would be rejected as being unnecessarily complicated. But the current voir dire process is byzantine enough that that hardly seems like a legitimate complaint.
edit (Tuesday, 9:16 AM): Landsburg has commented to this entry. In particular he points out that my assumption that juries would have the same accuracy in the arrangement with two half-juries as in the current system is incorrect; jurors would have more incentive to be accurate in his proposed system. This is true because in his proposed system the jurors are rewarded when both juries agree. But what I intended to show was that even without such a reward, his system still leads to a greater proportion of correct verdicts.
edit (Tuesday, 2:16 PM): Richard Dawkins suggested in 1997 that "Two juries of six members, or three juries of four members, would probably be an improvement over the present system". He also points out that jurors don't act independently, which is true; in my original analysis I was suggesting that even though jurors don't act independently, we'll assume that they act independently up until the moment they begin deliberation. This assumption is of course not true, but it was only a crude analysis.
During the lunch break I went to Reading Terminal Market, where I spent $9.05 for lunch (it was a large lunch, because I didn't want to be sitting around hungry); my pay for one day of jury duty was $9. (I was not chosen for a trial.) Coincidence? Probably, because I wasn't thinking "I'm spending my nine bucks on lunch" when I was walking around choosing where I would buy it.
Then I wandered down to a bookstore and found myself flipping through Steven Landsburg's book More Sex Is Safer Sex: The Unconventional Wisdom of Economics
. . (I didn't buy it; it seemed interesting, but not $26 worth of interesting.) In particular, this book suggests that the jury system is broken (the link goes to the Freakonomics blogs, where he was interviewed a few weeks ago). The basic idea is that jurors have no incentive to do a good job. This is clearly true, although I'm not sure how to incentivize the jury system. When I got home I ran across an entry in the Freakonomics blog which mentioned that book. Coincidence? Maybe. Maybe not. (And I thin I saw the original Landsburg interview a few weeks ago and forgot about it, which may have primed me to be more likely to look at that specific book.)Yet another coincidence: I went to high school with the judge's son. (I don't think this is how I got out of serving.) I also went to high school with the son of one of my panel-mates. It occurred to me as I was heading in this morning that if a few hundred people were called today, the chances I'd know one of them were not bad; there are probably about a million adults in Philadelphia, of whom I know a few hundred. I don't think anyone I know was there today (if so, I didn't see them) but as I said there were parents of people I knew. In some ways Philly is the largest small town in the country.
While waiting to be selected, it occurred to me that the voir dire procedure is set up so that no individual juror who is selected was biased. We were a panel of 50 for a sexual assault case, from which fourteen jurors were essentially selected; although I wasn't counting, I would guess that there were no more than twenty people who satisfied the following three conditions:
- 1. doesn't possess a strong technical background (we had to write our occupations on the forms that were distributed; it seemed that all the people who were asked about their occupation by the judge were either people in technical fields or people who worked as lawyers, police officers, etc.);
- 2. does not claim that jury duty would pose an extreme hardship;
- 3. had not been sexually assaulted or had someone close to them sexually assaulted. It's often said that one in four people are sexually assaulted during their lifetime.
Still, would it have been such a horrible thing to have a sexual assault victim on the jury? A randomly selected panel of fourteen would probably have had at least one. I don't see why each individual juror has to be unbiased in order for the group as a whole to be unbiased.
Finally, some math. In Landsburg's book he suggests the following: break each jury up into two half-juries of six. If they come to the same conclusion, that's the verdict; he wasn't clear on what to do if they came to opposing conclusions. (Presumably it would be treated like current hung juries are.) In this study by Bruce Spencer it's suggested that juries are "right" about 88% of the time. This got me thinking -- how likely does this mean an individual juror is to be "right" about the verdict? If we assume that jurors make their decisions independently, that majority rules (which is a bit ingenuous because juries in criminal cases have to be unanimous), and throw out 6-6 results, it turns out each individual juror has to come to the right decision with probability 63.6% to recover this 88% probability. This is related to the post I made a couple weeks ago about the World Series; if one team is slightly better than another, they have a decent chance of winning a single game but not so good a chance of winning a whole series. The teams here are, of course, "guilty" and "not guilty".
So what does this say for the half-jury suggestion? Let's say that each juror, indepedently, has a 63.6% chance of being right, and there's a jury of six. Say the defendant is guilty. Then the probability that all six jurors will say this is (.636)6 = .066; the probability that five think he's guilty and one thinks he's innocent is 6(.636)5(.364) = .227; the probability of the 4-2 split is 15(.636)4(.364)2 = .325. So the probability of one of these three results is 0.618; the probability of having a 3-3 split is 20(.636)3(.364)3 = 0.248. So the probability of a half-jury finding the defendant guilty is (.618)/(1-.248) = .821. Not surprisingly, this is less than the chance of a full jury finding the defendant guilty.
But the chance that both half-juries find the defendant guilty is (.821)2 = .674; the chance that they both find him innocent (even though he did it!) is (1-.821)2 = .032. So the probability of finding the defendant guilty, given that there's a verdict at all, is .674/(.674+.032) = .955. In the end, this plan achieves much greater accuracy at the expense of increasing the number of hung juries. It seems worth considering, though. (Incidentally, you can't beat the hung jury problem by changing the sub-jury size; either at least one sub-jury is of even size or there's an even number of sub-juries, since 12 is even.)
I suspect, though, that this sort of thing would be rejected as being unnecessarily complicated. But the current voir dire process is byzantine enough that that hardly seems like a legitimate complaint.
edit (Tuesday, 9:16 AM): Landsburg has commented to this entry. In particular he points out that my assumption that juries would have the same accuracy in the arrangement with two half-juries as in the current system is incorrect; jurors would have more incentive to be accurate in his proposed system. This is true because in his proposed system the jurors are rewarded when both juries agree. But what I intended to show was that even without such a reward, his system still leads to a greater proportion of correct verdicts.
edit (Tuesday, 2:16 PM): Richard Dawkins suggested in 1997 that "Two juries of six members, or three juries of four members, would probably be an improvement over the present system". He also points out that jurors don't act independently, which is true; in my original analysis I was suggesting that even though jurors don't act independently, we'll assume that they act independently up until the moment they begin deliberation. This assumption is of course not true, but it was only a crude analysis.
01 July 2007
37% "directly affected" by Pennsylvania minimum wage increase?
Pennsylvania's minimum wage was raised from $6.25 to $7.15 effective today.
According to the AFL-CIO, "37 percent of Pennsylvania workers who would benefit directly from a minimum wage increase work full time". The point that the AFL-CIO is trying to make here is that it's not just high-school kids working part-time who work for minimum wage.
This was misreported on myphl17 as that 37 percent of workers would be "directly affected" by the raising of the minimum wage. I take this to mean that 37 percent of Pennsylvania workers make between $6.25 and $7.14 an hour, which seemed ridiculously high. Of course, "directly affected" is vague, and might not mean exactly that. But it seems like the right interpretation.
The Keystone Research Center, in fact, said that "427,000 Pennsylvania workers would benefit directly from an increase in the state's minimum hourly wage from $5.15 to $7.15 by January 2007". The raise ended up happening in two steps, to $6.25 six months ago and then to $7.15 today; they further break it down to say that about 100,000 workers were making between $5.15 and $6.24, and 300,000 between $6.25 and $7.14. The population of Pennsylvania is about 12 million. So 2.5 percent of all Pennsylanians are "directly affected" by this raise; perhaps five percent of workers are.
Incidentally, I support raising the minimum wage; although I'm not sure what exactly it should be. The purpose of this post was to point out the numbers that were quoted that just didn't make sense.
According to the AFL-CIO, "37 percent of Pennsylvania workers who would benefit directly from a minimum wage increase work full time". The point that the AFL-CIO is trying to make here is that it's not just high-school kids working part-time who work for minimum wage.
This was misreported on myphl17 as that 37 percent of workers would be "directly affected" by the raising of the minimum wage. I take this to mean that 37 percent of Pennsylvania workers make between $6.25 and $7.14 an hour, which seemed ridiculously high. Of course, "directly affected" is vague, and might not mean exactly that. But it seems like the right interpretation.
The Keystone Research Center, in fact, said that "427,000 Pennsylvania workers would benefit directly from an increase in the state's minimum hourly wage from $5.15 to $7.15 by January 2007". The raise ended up happening in two steps, to $6.25 six months ago and then to $7.15 today; they further break it down to say that about 100,000 workers were making between $5.15 and $6.24, and 300,000 between $6.25 and $7.14. The population of Pennsylvania is about 12 million. So 2.5 percent of all Pennsylanians are "directly affected" by this raise; perhaps five percent of workers are.
Incidentally, I support raising the minimum wage; although I'm not sure what exactly it should be. The purpose of this post was to point out the numbers that were quoted that just didn't make sense.
the price of water
I found a link to this article in the New York Daily News which claims that in New York City, tap water costs 24 cents a gallon. This made me suspicious, because I'd always been under the impression that tap water was orders of magnitude cheaper than bottled water, and you're not going to find bottled water under, say, $1 a gallon. Also, you don't hear about people struggling to pay their water bills. This page (which was the first source I could find) says that with water-saving fixtures, a toilet flush is 1.5 to 3.5 gallons, and a shower is 2 to 4 gallons per minute. If it costs fifty cents to a dollar to run a shower for a minute -- and what's the average shower, ten minutes? -- you know people would consider not showering.
I was curious how much water costs in Philadelphia. (I rent; my landlord pays my water bill.) Anyway, it's $21.14 per 1000 cubic feet, or about 0.2826 cents (not dollars) per gallon, as Google's built-in calculator informed me.
And it turns out that New York City has new rates effective tomorrow; they're $2.02 per 100 cubic feet (see page 4), or 0.2700 cents per gallon. I suspect it was 0.24 cents per gallon before the increase, and someone just thought that looked wrong, because usually when you see, say, "Candy Bars: 0.75 cents" you just assume they meant three-quarters of a dollar. The difference is a factor of 100. I bet the person responsible for this error would say "oh, it doesn't matter". Okay, I'll cut their salary by a factor of 100. Let's see how they like living on a few hundred bucks a year!
Anyway, the Philadelphia Water Department has this fact sheet which includes the question: "How can I better understand the levels of elements in my water?" They go on to define parts per million/billion/trillion. They state:
It's 15.1 miles from Broad and Walnut to Conshohocken. (Note that the Google Maps route requires one to walk on the Schuylkill Expressway, which would be unwise.) This is 956,736 inches. A U. S. quarter is 0.955 inches across, so the distance is very nearly the length of a million quarters put side-by-side.
But then they go on to say:
Well, duh. But can you picture a barrel containing a billion red apples?
Let's say the inside of a SEPTA bus is eighty feet long, ten feet high, and ten feet across. Let's furthermore say that an apple is a four-inch sphere. Then if we fill the bus with apples, it's two hundred forty apples long, thirty apples high, and thirty apples across -- it holds 216,000 apples. So if we could fill five thousand SEPTA buses with apples, that would be a billion apples. That might be easier to picture. Except SEPTA only has 1,388 buses.. Okay, fill the busses with kiwis instead -- you can probably fit four kiwis in the space of one apple.
Finally, they say
Sixteen million miles is one-sixth of the distance to the sun! However, ten billion dollars actually might be an understandable figure. The per capita income in Philadelphia was $16,509 as of the 2000 census; the population of the city at that time was 1,517,550. The product of those is about 25 billion dollars. So a part per trillion is like finding a penny in all the money Philadelphians make in five months.
and yet... a part per trillion still is still very nearly a trillion molecules per mole, since Avogadro's number (6.02 × 1023) isn't far from a trillion trillion. A mole of water is 18 grams -- roughly a mouthful. How would people feel, knowing that there are as many molecules of [insert nasty contaminant here] in a mouthful of water as pennies made in Philadelphia in five months? (Yes, I know that I'm sweeping a lot under the rug here. My point is that a mole is a large number of large numbers.)
I was curious how much water costs in Philadelphia. (I rent; my landlord pays my water bill.) Anyway, it's $21.14 per 1000 cubic feet, or about 0.2826 cents (not dollars) per gallon, as Google's built-in calculator informed me.
And it turns out that New York City has new rates effective tomorrow; they're $2.02 per 100 cubic feet (see page 4), or 0.2700 cents per gallon. I suspect it was 0.24 cents per gallon before the increase, and someone just thought that looked wrong, because usually when you see, say, "Candy Bars: 0.75 cents" you just assume they meant three-quarters of a dollar. The difference is a factor of 100. I bet the person responsible for this error would say "oh, it doesn't matter". Okay, I'll cut their salary by a factor of 100. Let's see how they like living on a few hundred bucks a year!
Anyway, the Philadelphia Water Department has this fact sheet which includes the question: "How can I better understand the levels of elements in my water?" They go on to define parts per million/billion/trillion. They state:
1 part per million is similar to making a line of quarters from Center City to Conshohocken, and then walking that line to find the one quarter that is flipped up heads instead of tails.
It's 15.1 miles from Broad and Walnut to Conshohocken. (Note that the Google Maps route requires one to walk on the Schuylkill Expressway, which would be unwise.) This is 956,736 inches. A U. S. quarter is 0.955 inches across, so the distance is very nearly the length of a million quarters put side-by-side.
But then they go on to say:
1 part per billion is equal to 1 green apple in a barrel containing 1 billion red apples.
Well, duh. But can you picture a barrel containing a billion red apples?
Let's say the inside of a SEPTA bus is eighty feet long, ten feet high, and ten feet across. Let's furthermore say that an apple is a four-inch sphere. Then if we fill the bus with apples, it's two hundred forty apples long, thirty apples high, and thirty apples across -- it holds 216,000 apples. So if we could fill five thousand SEPTA buses with apples, that would be a billion apples. That might be easier to picture. Except SEPTA only has 1,388 buses.. Okay, fill the busses with kiwis instead -- you can probably fit four kiwis in the space of one apple.
Finally, they say
1 part per trillion is similar to 1 inch in 16,000,000 miles or 1 penny in 10,000,000,000 dollars.
Sixteen million miles is one-sixth of the distance to the sun! However, ten billion dollars actually might be an understandable figure. The per capita income in Philadelphia was $16,509 as of the 2000 census; the population of the city at that time was 1,517,550. The product of those is about 25 billion dollars. So a part per trillion is like finding a penny in all the money Philadelphians make in five months.
and yet... a part per trillion still is still very nearly a trillion molecules per mole, since Avogadro's number (6.02 × 1023) isn't far from a trillion trillion. A mole of water is 18 grams -- roughly a mouthful. How would people feel, knowing that there are as many molecules of [insert nasty contaminant here] in a mouthful of water as pennies made in Philadelphia in five months? (Yes, I know that I'm sweeping a lot under the rug here. My point is that a mole is a large number of large numbers.)
Subscribe to:
Posts (Atom)
