Yes, it is possible to anonymously aggregate votes from a set of public keys, and ensure that no key has voted twice.
It's also possible to ensure that one's own vote was included in the total.
The fact that this is even possible is deeply un-intuitive as it requires some of the most sophisticated cryptography.
That's probably the greatest barrier to adoption.
When people think of electronic voting, they think about trusting a company to make machines that operate on plaintext, and require humans to guard access to the machines.
They aren't thinking about systems that are provably correct, where it is more likely for an asteroid to wipe out the country conducting the election than for the election results to be incorrect.
For the details and tradeoffs, I highly recommend Vitalik's blog.
The problem is ensuring that the set of allowed public keys you have is actually the set of allowed public keys you want.
As others in the thread have said, there's nothing stopping the government from manufacturing millions of fake passport (or even just generating millions of fake passport keys) and using them to rig the election.
For the purposes of this, I was assuming 1:1 passport to citizen and just wondering if that can be made anonymous. The real idea with an untrusted passport authority doesn't work ofc.
The fact that this is even possible is deeply un-intuitive as it requires some of the most sophisticated cryptography. That's probably the greatest barrier to adoption. When people think of electronic voting, they think about trusting a company to make machines that operate on plaintext, and require humans to guard access to the machines. They aren't thinking about systems that are provably correct, where it is more likely for an asteroid to wipe out the country conducting the election than for the election results to be incorrect.
For the details and tradeoffs, I highly recommend Vitalik's blog.