It gets it wrong because the current "AI" for filling out forms is extremely weak and brittle compared to the general language models we have now.

Language models seem pretty weak and brittle in my interactions with them too.

Do you have an example form field that a general language model could fill out better than a human + highly focussed deterministic algorithm?

Ecommerce checkout. Filling out my address, billing adress, and credit card information. Things like drop downs or different formatting can mess up the current basic ones, but it really shouldn't be that hard for AI to figure out how to fill out such information it knows about me into the form.

I think I've found those unreliable in the past, but much more reliable as time goes on. I can't really remember the last time an address or credit card info was mishandled by autofill. I get that addresses can be poorly defined, but for one you've entered yourself, that you just want to be re-entered, I don't see why we can't solve that problem without AI.