you are going the right direction with your swing.,Real good game,,very simple, easy to fix...
Take it just a step further.
Use the left hand a little more (for weight balance) and get your push away to go toward the center of your body...Just a little..By using the left hand you will free up the swing a little easier and by this you will not let the ball get to far away from your body on the down swing...
A couple things will get fixed by this simple trick..
You walk around the ball good so this will be easy for you to do.