I cut Switch Outers down with a router sometimes too. Speed is a little high, would actually melt the plastic a little. I used a Dremel tool on the plastic to make it flush with the ball, and the the bevel sander.
Press speed could be a little high, but that is a BIG bit spinning in there, and if you are just shaving some slug out with the 1 1/2 bit, you probably will get some vibration/movement.
Also, I bought an 1 1/2 end mill from Turbo. Cost about $150 if I remember correctly, but if you run this bit down through the hole after the 1 1/2 carbide drill bit, it will make the hole perfect all the way to the bottom. The outer sleeve will slide in sweet, and you won't have as much problem with the inner sleeves not turning easily in the outer sleeve. A wise investment for you in my opinion.
Be careful with the end mill, that is a big azzz bit, I used WD40 on it while drilling when it was brand new. Almost too sharp.
--------------------
Lane #1