Judy,
In addition to the other responses you’ve received, consider using a
bushing with your drill press. Make a holding fixture to hold the
bushing on top of the material that you are drilling through. The ID
of the bushing should be just slightly larger than the drill bit, and
this ensures that the bit stays in position as it bites into the
material. This will significantly help the “walking” that you’re
experiencing.
Also, with the material thickness you’re working with compared to the
hole size, be careful about building up heat on your drill bit tip.
With a bushing you’re able to “peck” at the hole, thus avoiding heat
buildup.
Even with a bushing, you may not get the accuracy that you want with
a drill bit. Consider using a smaller bit and finish the hole with a
reamer, with a bushing guiding each one. Reaming the hole instead of
drilling it will give you more accuracy. You could even consider an
end mill, but I doubt that you can get one as small as you are
looking for. Good luck… this task isn’t for the faint of heart.
Jamie