I just wondered -- what is the typical age of a PhD recipient? A bit of Googling turned up this table from Inside Higher Ed, which conveniently sorts by discipline; it reports on an NSF brief. Mathematics and physics are tied for second lowest median age at 30.3; chemistry is the only discipline that's lower, at 29.6.
The table I linked to also gives the median time from getting the bachelor's degree to getting the PhD; by subtraction one can get some number that is a "typical" age of bachelor's degree receipt for students who eventually get a PhD. The median time from bachelor's degree to PhD in mathematics is 7.9 years. Subtraction, 30.3 - 7.9, gives 22.4 as a "typical" age (the difference of medians, which isn't really meaningful) for students getting a bachelor's degree who eventually go on to get a PhD in math. (The highest typical age at bachelor's degree is 25.3, for people getting PhD's in education.) This is the minimum among all eighteen disciplines covered here. It's hard to imagine a median much lower than that given the age at which students typically enter formal education and the number of years it takes.
I interpret this as saying that students who get PhD's in mathematics are less likely to take time away from formal education between high school and college or to take longer than the traditional four years to graduate from college. I'd be interested to see if this is because students who spend time away from formal education "lose" whatever mathematics they knew and have trouble picking it back up again; it's a popular conception that mathematics is more "hierarchical" and so this is more of a problem there than in other fields. (Not having much experience with other fields, I can't say.)
Also, chemistry has a median registered time to degree (time from entering a doctoral program to receiving the PhD) of 6.0 years; the next lowest is mathematics at 6.8. Why is chemistry such an outlier?