Just from my own personal experience, the problem isn't so much that they reach those temperatures, its that they reach those temperatures and then you bring them into a much warmer or cooler climate and then immediately toss one.
I left an El Nino 2000, Storm Eraser, and a Columbia 300 Beast all in my parents barn for 3-4 years, brought them in, let them warm to room temperature and had zero problems.
The core, filler and shell are all made of different materials and will have different expansion/contraction rates so if you take them from extreme cold to extreme warm you are probably just increasing the chances that it will crack.