You could run into a problem with how much top or bottom weight the ball has. Maybe if the ball has higher-than-normal starting top weight, removing a few extra ounces won't hurt you statically, but if it's got normal top or low top and you put in a weight hole too, removing more weight from the thumb and fingers could (could) result in too much bottom weight.
Probably that wouldn't happen as you're allowed up to 3oz of top/bottom weight after drilling, but you never know.
I doubt you could make more than a 1-2oz difference anyway.
Have you tried sanding that extra ounce or two off? I read a post a while back about a guy who sanded his 16# ball down to 15# because the 16# was too heavy.
SH