Nassim Taleb (author of The Black Swan and Fooled By Randomness, both of which I enjoyed very much), takes on Edge's question for 2008: "What have you changed your mind about?"
His short answer is "The irrelevance of 'probability'". Naturally, I took offense.
It turns out that Taleb is not saying that the theory of probabilities is worthless for real-world situations -- although this would be the sort of thing he would say, and as he has large piles of money which have been made in the markets, perhaps he knows something I don't. It could be the case that probability theory, like any other axiomatic theory, just doesn't reflect what happens in real life. Compare Newtonian mechanics or Euclidean geometry; who will be probability's Einstein or Riemann?
Rather, Taleb is saying that we need to take into account expected value -- the product of probability and payoff. And as I understand it, the funds he manages lose a small amount of money most days -- but they make very large amounts of money on the days they make money, so they come out in the black. (Psychologically, this is hard for human beings to take, because basically we feel sad if we lose money and happy if we make money, and we're not that sensitive to the amounts lost or gained; thus on most days someone with this sort of scheme feels sad. The trick, apparently, is to not look at your portfolio that often; most days you lose, but most months (say) you win.)
More generally, depending on the problem, knowledge of the whole distribution of some random variable can be useful. Of course we can't cut a whole distribution down to a single number. Or even two numbers. Not every distribution is normal.
(Read the rest of the responses to this question, too. A hundred smart people have things to say.)