Personally, if you want to swing the ball, drifting is mandatory. Maybe not as much as you currently are doing. You will not be very consistent with your release if you prevent a fluid pendulum swing. If you are perfectly square at the line, then it is difficult to get the ball to follow a path that isn't straight up the lane. You want your shoulders square with your target, and to accomplish that you have to slide toward your target. It doesn't require much drifting but it will make you more consistent.
I will drift left first (RH) then back to the right toward my target when I get to the line. My only problem is when I am all the way left and the ball return starts to get in my way. I cannot think of one professional that plays a lot of area not doing this. But then again, I don't really look at how much they drift. It could only be a couple of boards and it would be hard to pinpoint.
Just as long as you can be consistent (one board adjustment at your start equals one board at the line) and you drift the same every time, you should be ok.
--------------------
I am not a pro-bowler, but I do play one on BallReviews.com