Counterexamples in Probability And Statistics (Joseph P. Romano and A. F. Siegel) and Counterexamples in Probability and Real Analysis (Gary L. Wise and Eric B. Hall) both seem to be books in the tradition of Counterexamples in Analysis (Bernard Gelbaum and John Olmsted) and Counterexamples in Topology (Lynn Arthur Steen and J. Arthur Seebach. These are books that collect the examples just "outside" the boundaries of the various standard theorems, the point being to explain why one needs the seemingly strange collections of hypotheses that seem to begin every analytic theorem. (Hence the tags "education" and "teaching"; I've often seen these counterexample books described as "anti-textbooks", and as being complementary to standard textbooks which often spend most of their time telling you what's true.)
It seems that these books are concentrated on the analytic end of mathematics; I couldn't find, for example, books of counterexamples in algebra, combinatorics, or number theory. There is, however, Theorems and Counterexamples in Mathematics. My sense is that the nonexistence of these books is connected to the fact that those fields don't seem quite as rife with theorems where all the work is hidden in the definitions.