Back in ye olde times a half hour or so with a chalk board and some smart people, go away try some things then come back and erase / write some more worked pretty well.

This is more closer to throwaway code than design docs.

Why not both? And it can be done productively: start coding (prototyping) if planning is difficult and vice versa.

That rests on the assumption that you'll notice that planning is "difficult". Whereas what I've usually found is that people build elaborate plans and feel like they've made progress doing so, only to discover when they come to implement them that there was a false assumption in step 1 that renders everything else pointless (or, worse, the plan should really be fundamentally reconsidered in the light of this info, but you've spent all this time on it, and it's not outright impossible to keep following it, so...).

An extension of this is what I call problem solving by differentiation. People who don't understand the problem space defer the part they don't understand by creating premature abstraction upon premature abstraction. In their head they seem to be slicing up the problem until there is none left, but in practice this continues until they finally understand the core domain problem or someone who does gets assigned. But those layers of abstraction don't die...

I thought this is a solved problem. Steps:

1. Have some long term goals.

2. Design a small iterative improvement or a minimum viable product that should bring you closer to your goals.

3. Build just that.

4. Reevaluate.

I.e. I thought the idea is to embrace changing the approach or even pivoting when necessary.

It is. But for some reason a lot of people prefer to spend time planning and even moralising about how virtuous they're being by doing so. Frankly I'm convinced it's more of a cultural, Protestant-work-ethic type issue - you can't reason people out of planning because they never reasoned themselves into it in the first place.

Came here to say this.