I’d definitely stick with the pumice. I screwed up a rather nice amethyst cab with a pink silicone wheel by not realizing I’d brushed up against it, and the better part of a year later I’m still annoyed about it. I’m not sure about Tripoli- a quick google tells me Tripoli is 5-7 mohs, and your tourmalines can be at the upper end of that range too. If you’re working with a lot of them it might be worth (potentially) sacrificing a small one to run your preferred polishing compounds and wheels over it to see what happens.
Something I keep meaning to do is to compile a chart for myself of the wheels and compounds I use, and the hardness of a wide variety of stones. I do enough googling that my phone understands the word Mohs now but it would be nice to have it at a glance above my bench. If I ever actually get this together I’ll definitely post it in case others would find it useful.