Safe polish around colored stones

Hello,

After using a hammer to bezel set softer colored faceted stones like
topaz, peridot, garnet etc, what would you suggest the best polishing
progression would be starting from the hammered texture left by the
tools all the way up to a high luster? I mostly work in sterling
silver at the moment.

Thanks for your input!
Chris