That seems a bit odd. I was having a similar confusion the other day so did a conversion and see that 0.1mm is about .004". Do the plans not say anything about this, like give a tolerance or some suggestion? Neither a normal drill press nor a hole saw have anything like that for a tolerance. In fact you can just look at most hole saws and visually see they are just a bit out of round, especially where they are welded ( hole saws are made from band saw material that is welded into a circle ).
If you buy a 33mm hole saw the actual hole is likely to be more than 34mm, maybe even 35mm.
I'm sorry I'm not really familiar with the Voodoo plans, what are you using for a diff?
At some point I bought a couple of the biggest files I could find and it takes just a few minutes to adjust something with them and they also make quick work of cleaning up sharp edges (dressing) after you cut a piece of stock. It's worth doing that and it will also adjust the size of a hole or the edges of something you notch.
Important, especially when notching with a hole saw you should cut back the sharp edges at the points because you don't want to weld the flimsy edge, you want the full thickness of the tube where you weld around it. So file those edges back a bit to get to real metal.