A bit of confusion about annealing sterling silver

I was taught to avoid over-heating sterling silver during annealing

  • i. e.

taking it beyond the “dull red” stage. My understand was that going
above this temperature would be counter-productive in that it would
make the metal a bit harder or perhaps more brittle.

So let’s say I’m melting some scrap to make balls/granules, and that
I want to flatten those balls into “patties” or stamp them into a
shape using a shot plate. Obviously, since I melted them, I went WAY
beyond the “dull red” stage. So should I anneal them before working
them?