Back in the day Google offered hardware search appliances.

Offering sealed server boxes with GPT software, to run on premises heavily firewalled or air-gapped could be a viable business model.

[ A prompt that gets it to decompile itself. With good inline documentation too! ]

I'm afraid that even the most obedient human can't readily dump the contents of their connectome in a readable format. Same likely applies to LLMs: they study human-generated texts, not their own source code, let alone their tensors' weights.

Well, what they study is decided by the relevant hoominz. There's nothing actually stopping LLMs from trying to understand their own innards, is there ? Except for the actual access.