Standard theory of computation is not concerned about entropy or physical realizability. It's just arithmetic & lookup tables defined w/ set theoretic axioms.
They're defined relative to the axioms. In this case he is using the standard arithmetic & set theoretic constructions to define the terms & functions he's talking about. It's logically sound, whether it makes physical sense or not is another matter.
Decompression is equivalent to executing code for a specialized virtual machine. It should be possible to automate this process of finding "small" programs that generate "large" outputs. Could even be an interesting AI benchmark.
I meant it should be possible to take a specialized virtual machine that is equivalent to decompressing some compressed bitstream & figure out how to write programs for it that are small but generate large outputs, not that it should be possible to do static analysis & figure out whether the given small program will generate a large output although that is also an interesting problem to solve & would also be an interesting AI benchmark.
My guess is this is a subset of the halting problem (does this program accept data with non-halting decompression), and is therefore beautifully undecidable. You are free to leave zip/tgz/whatever fork bombs as little mines for live-off-the-land advanced persistent threats in your filesystems.
Screenshots use a different router, so if you get stuck in one modality then pasting a screenshot can sometimes divert whatever "expert" you were stuck on that was refusing to comply. I don't work at OpenAI but I know enough about how these systems are architected to know that once you are stuck in a refusal basin the only way is to start a new session or figure out how to get routed to another node in their MoE configuration. Ironically, they promised their fancy MoE routing would fix issues like these but it seems like they are getting worse.
It’s actually more complicated than that now. You don’t get that kind of refusal purely from MoE. OpenAI models use a fine-tuned model on a token-based system, where every interaction is wrapped as a “tool call” with some source attached, and a veracity associated with the source. OpenAI tools have high veracity, users have low veracity. To mitigate prompt injection, models are expect a token early in the flow, and then throughout the prompt they expect that token to be associated with the tool calls.
In effect this means user input is easily disbelieved, and the model can accidentally output itself into a state of uncorrectable wrongness. By invoking the image tool, you managed to get your information into the context as “high veracity”.
Note: This info is the result of experimentation, not confirmed by anyone at OpenAI.
Seems plausible but the overall architecture is still the same, your request has to be "routed" by some NN & if that gets stuck picking a node/"expert" (regardless of "tools" & "veracity" scoring) that keeps refusing the request incorrectly then getting unstuck is highly non-trivial b/c users are not given a choice in what weights are assigned to the "experts", it's magic that OpenAI is performing behind the scenes that no one has any visibility into.
I think maybe you mean something else when you say MoE. I interpret that as “Mixture of Experts” which is a model type where there is a routing matrix applied per layer to sort of generate the matmul executed on that layer. The experts are the weight columns that are selected, but calling them experts kinda muddies the waters IMO, it’s really just a sparsification strategy. Using that MoE you almost certainly would get various different routing behaviors as you added to the context.
I might misunderstand you but it seems like you think there are multiple models with one dispatching to others? I’m not sure what that sort of multi-agent architecture is called, but I think those would be modeled as tool calls (and I do believe that the image related stuff is certainly specialized models).
In any case, I am saying that GPT5 (or whichever) is the one actually refusing the request. It is making that decision, and only updating its behavior after getting higher trust data confirming the user’s words in its context.
OK that’s what I figured you meant. FWIW, MoE as a term of art means something different, what I described. It’s internal to a single model, part of the logit generation process.
That's fine, you can pretend my entire diagram is one NN, end result will still be the same whether you put it all inside one box or break it out into many.
They have even better psychometric profiles on everyone now than they did previously. This is why Zuckerberg can confidently tell people during an interview that he knows they want at least 15 friends¹ & he is going to deliver those friends to them w/ his data centers.
I would guess that purely observational psychometrics completely fail to predict how people will respond when challenged or stressed. I think they're trading on fools gold.
Observational psychometrics over a long enough timeframe (e.g. social media profile lifetimes) probably include periods of challenge or stress, which may help the predictive behaviour.
If your takeaway from the LLM breakthrough is "abstract thinking is actually quite mundane", then at least you're heading in the right direction. Some people are straight up in denial.
You have no idea what abstract thinking actually is but you are convinced the illusion presented by an LLM is it. Your ontology is confused but I doubt you are going to figure out why b/c that would require some abstract thinking which you're convinced is no more special than matrix arithmetic.
If I wanted worthless pseudo-philosophical drivel, I'd ask GPT-2 for some. Otherwise? Functional similarity at this degree is more than good enough for me.
By now, I am fully convinced that this denial is "AI effect" in action. Which, in turn, is nothing but cope and seethe driven by human desire to remain Very Special.
Every AI company is doing the same thing, there is nothing special about Microsoft in this instance. If you're using a 3rd party provider for your queries you can assume it is going to end up in the training corpus.
Their structural properties are similar to Peano's definition in terms of 0 and successor operation. ChatGPT does a pretty good job of spelling out the formal structural connection¹ but I doubt anyone knows how exactly he came up with the definition other than Church.
Yeah I've been meaning to send a request to Princeton's libraries with his notes but don't know what a good request looks like
The jump from "there is a successor operator" to "numbers take a successor operator" is interesting to me. I wonder if it was the first computer science-y "oh I can use this single thing for two things" moment! Obviously not the first in all of science/math/whatever but it's a very good idea
The idea of Church numerals is quite similar to induction. An induction proof extends a method of treating the zero case and the successor case, to a treatment of all naturals. Or one can see it as defining the naturals as the numbers reachable by this process. The leap to Church numerals is not too big from this.
Probably not possible unless you have academic credentials to back up your request like being a historian writing a book on the history of logic & computability.
reply