Rivets bend like that, not because they may have been annealed, but because of several variables. One, the rivet may have been too long. Two, the rivet did not fit snugly into the hole(s) through which it was inserted. Annealed rivet wire is easier to upset (the tapping down of the rivet) than work hardened wire, so don't worry about the fact that the rivet wire may be annealed. Also, a properly upset length of rivet wire becomes slightly work-hardened during the process of upsetting it.
THE LENGTH OF THE RIVET
The length of a rivet is determined by the diameter of the rivet wire. A good rule for rivet length is that the rivet should not extend beyond the exit from the hold more than ca. 1/2 the thickness of the rivet wire. You don't need calipers to measure this, just eyeball it. It's a little more difficult when you are joining softer materials (like leather to metal), because the material tends to compress, but you can take this into account as you trim your rivet wire to the correct length.
THE HOLE FOR THE RIVET
The hole for the rivet must match the diameter of the rivet wire, as closely as possible. To achieve this accurately, it helps to use a drill gauge. Insert the rivet wire into the hole in the drill gauge into which the rivet wire fits snuggly. Choose a drill bit that is the size that fits into the same hole. I generally will choose a drill bit that is close to that size, but a little larger, just so I don't have to struggle inserting the rivet. Rule of thumb, though, is to choose the drill bit to match the diameter of the rivet exactly. If that hole tends to resist the insertion of the rivet, you can very slowly enlarge that opening using a round needle file or a very small steel cylinder bur. Alternatively, if the hole is too snug for the rivet wire, try waxing the rivet with beeswax, which will help the rivet slide through the several openings. Hope this helps.