The nicest cuff links don't use a spring toggle. they have an ornament on both sides of the cuff connected by a curved bar or a hinged bar or a bit of chain. Ones connected by chain can be hard to put in.
If the ornament on the outside of the cuff is too big to go through the buttonhole you can make the inside one a bit smaller. And they don't have to be identical. The inside and outside designs can be complementary. If you want both sides of the link to be large, they can be connected with a bit of spring gold bent into an oval or peanut shaped clip.
Here's a simple pair of onyx and diamond links connected by a bit of spring gold.
Here's a pair connected by chain, with the halves complementary rather than identical.