Fine silver will harden and the grain boundaries are the issue- in a pure (relatively speaking versus common alloys) you will have a point of hardness that is the max for that metal. The issue lies in the grain boundaries and the effect of the crystals slipping along those lines. Annealing is the process of “reliving” the stress introduced by manual compression of those grain boundaries.
Shallow surface hardness can be achieved with a tumble with stainless steel shot which will burnish and compact the surface somewhat- but the core will still be relatively soft. (and the surface finish/details or texture may suffer due to the impacts of the shot)
To obtain a hard core to the item- hammering on a steel or hard block and then moving to final form with files, saw and hand tools will not remove the introduced compressive stresses.
Work hardening in your original question is relative. Fine Silver “will” work harden compared to an annealed piece of the same silver.
Your task is to identify what hardness you “need” to achieve for your end-use and then find the alloy that will match the use. I would look into the British coin formulas/alloys used over the years as theirs are well researched and written about and see why each was used, abandoned or adopted. Coins were soft in the “as struck” states and there was a need to make the coins last longer as well as show signs of tampering. (Although the tamper proof designs were not related to hardness- only fairness of commerce)
Hope this helps- not trying to be long winded. Nerd Alert- If you have a Engineering Library near you or a higher end academic library- Look into the ASM Handbook Vol 9 by George Vander Voort- it has a good section on alloys of precious metals- please dont buy this monstrous tome- its WAY too full of nonsense for jewelry work (unless you like to be the nerd and plan on doing casting/heat treating on an industrial scale)