I also do a lot of inlay. I mostly use a flat lap with resin/diamond bonded disks. I used said disks for polishing softer stones like turquoise and lapis.
I haven’t noticed any silver contamination on the stones but I rarely cut jade.
If I’m doing an inlay piece with harder stones like jasper, agate, petrified wood, etc. then I’ll use the 8" wheels for the roughing work down to 180 grit then finish on the flat lap.
Rocky Mountain Wonders