After hitting it with 500 I would've applied very light pressure with the 4000 pad for 3-5 seconds instead of 10-20.
It's always easier to go back and hit it again with a 4000 pad if you didn't hit it enough the first time. Hit it for too long and you make it shinier and lose the effect the 500 pad had as well which means you have to start the process all over again.