Without a video it's hard to say, but a bad pushaway direction will lead to a bad position at the top of the swing. This will force you to "redirect" the ball at the bottom to get it on line. When you are playing straighter angles this won't be as much of an issue, but inside angles are tough to keep consistent.
I fought this for years because my swing would get outside of my body at the top, this made me miss inside more often than out.
--------------------
Little known fact: In Russian "Hope" and "Change" translate to "Tax" and "Spend"