By hand, you shouldn't do much damage, but why get into such a habit. Use a cloth and good cleaner (Track's Clean and Dull or its newer version, Clean and Tacky, is just about the best), because roughed up that way, a particle ball like the Goliath absorbs and incredible amount of oil VERY rapidly. In fact, you'd better wipe the ball after every throw with a micro-fiber towel.
I'd do a hand green pad job every 3-5 3-game sessions, if I were you.
--------------------
"None are so blind as those who will not see."