Another solution I'd think about is having some small collars made to fit in the larger hole at the top.
Have them made to tap in with a bit of bearing fit.
This way you can keep everything else standard.