SSNs are probably the closest existing parallel. How many people in the US of voting age haven't had their SSN leaked yet? Obviously SSN is abused in the things it is used for (I.e. identification), but I don't see any indication voter IDs wouldn't be abused the same way.

SSN is a (mostly) permanent number you use for many many things. A voter ID would be unique with every upcoming vote. Election specific public key would probably be a more correct term.

Abuse of it would definitely be bad, but it would be more difficult as you'd have to repeat the identity theft every election, rather than knowing someone's lifetime secret number.