So one of the courses I'm teaching this semester is introductory probability, with a calculus prerequisite. Not surprisingly this attracts a lot of students in various subjects that are math-heavy but that are not actually mathematics or statistics. They also tend to be juniors and seniors.
In office hours today, one of my students got to criticizing the fact that lots of consulting, finance, etc. jobs like to ask questions like "how many ping pong balls fit in a 747" in interviews, on the basis that this has nothing to do with what they'd actually have to do on the job. I pointed out that there's some difficulty in knowing whether your hiring practices are working. You'd like to compare the success of the people that you hire into your company with the success that the people that you don't hire would have had in your company. The former might be difficult to measure, but it's probably not impossible assuming your company does something quantifiable. But the latter is essentially impossible.
So what's the right way to design this sort of study? Is there any serious research on which interview techniques actually succeed in finding the best people?