I think it would be fine to remove some material. You'd be removing so little, I doubt there would be any structural consequences. Removing with a dremel or similar is NOT what I would do. You will not be able to make a nice hole. It's very hard to control how much you'd be removing with a rotary tool, unless you're an experienced dentist.
A fixed hole saw of the diameter you need, an adjustable hole saw, or a concave file would be better alternatives. If a hole saw is used, either do it with a drill press, or clamp a circular template in place so that the hole saw doesn't move around. If you don't have a drill press, it should cost about $10 to have a carpenter make one for you.
