From Lance's Blog
From Lance's Blog
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore’s law, or rather its generalization of continued exponentially falling cost per unit of computation.” - Richard Sutton
For years, after diving into the field of AI and trying to optimize the models into somewhat General-like(e.g Gaussian Process, Bayesian Inference, SLMs) the ego-inflation became massive with no actual progress. Papers and Companies argue why some architecture should be awsome (e.g JEPA) and a road to AGI.
But, we fall into the same hole once again. As Geoffrey Hinton said in his latest interview, we don’t know why these machines actually work. Every model should fall into a local-minima, and cannot go beyond the bias-variance tradeoff. But, attention and residual networks goes through every breakthrough.
yann lecun's talk
Anti-LLM researchers usually argure about two things. 1) LLM’s need too much information(data) to learn and 2) LLM’s cannot search. But in my opinion both are wrong. The metaphor between LLM and a child miss the information density, and the searchability of LLM in Chain-of-Thought is 100% true(which will be discussed in another post). The emergent capability of LLMs never seems to stop.
But after the astonishing success, there seemed to be a never-ending stop towards this machine’s capability. Spatial Reasoning, millennial math problems, Robotic manipulation, Machanical Engineering … It was glorious at first, but came as a disaster to what we humans should do in this Super-Intelligence-ish machine.
Now maybe we should abandon inductive-bias.
From François Chollet (@fchollet), Oct 2, 2026
The inductive <-> transductive transition made searching at test-time more important. How can we design a search-space more suitable for AI?