No

When you change from inserts to no inserts do you use the same size hole as you use for grips.

The replies above are concerning using the same size hole as was used for placing the inserts into.

If you are asking about using the same size as the finger insert size for the drill bit, I am not sure.  I assume so but not positive.  Example, your finger inserts were sized 25/32 and 23/32, so without inserts, using those size drill bits to drill the finger holes.  Hopefully, someone with experience will answer that.

if you have your inserts tight; then you need a slightly larger hole.
even bigger if you plan to put tape in them

best way is to just measure your fingers instead of looking at your inserts.