There are definitely ways to prevent cracking, even with using interchangeable thumbs and finger grips. For the thumb, using a pilot hole before drilling the 1-1/2" (or whichever size is used) and letting the bit cut instead of plowing through it. Then, before installing the sleeve, take the sharp edge off. This works the same for slugs, when I was in the shop I would run into issues when my bits were getting dull and not drilling a clean hole. I would sometimes have to pound a slug in and that kind of pressure almost always led to the ball splitting over time. (Once on the press in my novice days)
With grips, making sure there's 1/4" bridge, touch the tops with bevel before installing grips. Same with extra holes, bevel them.