You could use some "quick-dry" plug and it would be ready on 20-30 minutes I think.
I'm old fashioned so "anything put together easily might fall apart easily". Shims could fall out or break. Tape, may be an option, but you might spend more time getting it "right" in the small finger hole area than plugging with the quick setting plug now available.