In addition to the factors charlest mentioned for maintenance, it's a performance issue.
The ball must find adequate friction in order to change direction and drive through the pins effectively. If it's covered with oil how would someone expect that to happen?
The shammy is an effective tool, but I have had great results with suede and synthetic suede "towels" and find them easier to use and maintain.
In order of effectiveness, in my opinion:
4- Terry towel - Least effective, but better than nothing. Smears oil, but will absorb a little of it.
3- Microfiber - Will pull more oil than the terry cloth. Very affordable.
2- Suede (micro-suede, etc.) A less expensive option than shammys and very effective in my own testing
1- Shammys - Very effective, but the most pricey option. (Check your bowling center's lost and found because these get left behind, making them even more expensive.)