How to know if your hiring practices work?

So one of the courses I'm teaching this semester is introductory probability, with a calculus prerequisite. Not surprisingly this attracts a lot of students in various subjects that are math-heavy but that are not actually mathematics or statistics. They also tend to be juniors and seniors.

In office hours today, one of my students got to criticizing the fact that lots of consulting, finance, etc. jobs like to ask questions like "how many ping pong balls fit in a 747" in interviews, on the basis that this has nothing to do with what they'd actually have to do on the job. I pointed out that there's some difficulty in knowing whether your hiring practices are working. You'd like to compare the success of the people that you hire into your company with the success that the people that you don't hire would have had in your company. The former might be difficult to measure, but it's probably not impossible assuming your company does something quantifiable. But the latter is essentially impossible.

So what's the right way to design this sort of study? Is there any serious research on which interview techniques actually succeed in finding the best people?

The mathematician and the lumberjack

You remember the whole "mathematicians have the best job in America" survey from jobsrated.com (which was based on some fairly questionable criteria)? Well, apparently Sean Hurley at NPR did a story a couple weeks ago, where he talked to a mathematician (Peter Winkler, of Dartmouth) and a lumberjack. Peter Winkler likes his job. So does the lumberjack, although the pay sucks and trees occasionally fall on you, which sucks too.

We're number one!

A ranking of 200 jobs in the United States. Mathematician, actuary, and statistician are #1. #2, #3. Lumberjack, dairy farmer, and taxi driver are #200, #199, and #198. ("Farmer" and "chauffeur" are a bit higher on the list.)

Their one-sentence description of what a mathematician does is "Applies mathematical theories and formulas to teach or solve problems in a business, educational, or industrial climate", which isn't too bad. The word "formulas" in there rubs me the wrong way, though; I don't like the implication that mathematics is all about formulas, and I tend to use "formulae" as the plural. Here's an explanation of the methodology. They don't give the scores they assigned to mathematicians on each of the many factors they take into account. But basically, it's a good job because we sit on our asses, don't have to deal with customers, and make good money. (Well, I suppose I make good money if you factor in the fact that I don't pay tuition...)

Many of their top jobs are one what might call "academic" jobs -- the top ten also includes statisticians, biologists, historians, and sociologists. I'm wondering whether it's really true that academic jobs are generally good, or whether their methodology consistently overrates such jobs.

Via Not Even Wrong (Peter Woit); see also the Wall Street Journal and reddit.

Edit, 5:56 pm: see also See also Cosmic Variance (which, as usual, has many commenters, some insightful), Rigorous Trivialities, Unapologetic Mathematician.

Edit, Friday, 9:47 am: See also The Accidental Mathematician and Junk Charts.

Edit, Tuesday 1/13, 12:05 pm: And Computational Complexity.