I didn't have the problem you had (mis-sized holes), but one of the reasons I stopped using inserts was because I was between two sizes and used to oscillate between the two closest ones.
Even after discovering Vice brand half-sizes, I still couldn't settle on one size and stick to it. Since I switched to no-inserts, each ball (at least at the fingers) felt identical. Had other issues, but at least without the inserts, I can't constantly adjust them back and forth in my OCD (Obsessive Compulsive Disorder) way. No... I have not been diagnosed with it, but anyone observing the constant tweaking/adjusting I do with bowling equipment, would surely label me as such.
And surprisingly.... my fingers which had started to swell and shrink somewhat, have never done that since switching to no-inserts. Not sure why, but added benefit has been consistent feel in the fingers. Only the thumb size has been a continuing problem in the 2 years since I stopped using inserts.