I would say the main reason it's hard to follow abstract maths lectures by listening is not the difficulty of the concepts per se, but simply because it's so visual: it relies on notation and diagrams
But mathematicians can talk to each other about arbitrarily abstract concepts, as long as they have enough shared background, and they don't (always) need a blackboard to do it.
Conversely, you can have conceptually very simple things that are basically impossible to follow just by listening, like multiplying two nine-digit numbers or following one of Euclid's proofs in plane geometry. The difficulty isn't about abstraction, but how many things you have to hold in working memory
But mathematicians can talk to each other about arbitrarily abstract concepts, as long as they have enough shared background, and they don't (always) need a blackboard to do it.
Conversely, you can have conceptually very simple things that are basically impossible to follow just by listening, like multiplying two nine-digit numbers or following one of Euclid's proofs in plane geometry. The difficulty isn't about abstraction, but how many things you have to hold in working memory