The world of Large Language Models (LLMs) is advancing rapidly, with new breakthroughs and innovations emerging every day. But amidst the excitement, it’s essential to take a step back and separate fact from fiction. What do we really know about LLMs, and what do we still need to figure out?
What We Know
We’ve made significant progress in understanding the inner workings of LLMs. For instance, we know that scaling laws are evolving, and strategic allocation of inference computation can make models 14x smaller while maintaining performance. We also have a good grasp of the Transformer architecture and the importance of mechanistic interpretability.
Moreover, we’ve seen impressive advancements in specific domains like medical exams and legal reasoning, where models can achieve expert-level performance. And, we’ve developed scalable evaluation methods, such as using one LLM to evaluate another’s output.
What We Don’t Know
Despite these advances, there are still many unanswered questions. Why do LLMs develop complex reasoning abilities when trained to predict the next word? What is the true nature of ’emergence’ in LLMs? And, how do we prevent models from generating false information, or ‘hallucinations’?
We also struggle to distinguish between systematic generalization (true reasoning) and interpolation (pattern matching). And, we need to better understand how models integrate information and perform complex, multi-step reasoning.
Why We Argue About This
The LLM community is not immune to disagreements and debates. Methodological disputes, semantic slippage, and the capability vs. understanding gap often lead to philosophical rather than scientific debates. The fact that many capable models are proprietary also limits our ability to verify claims independently.
TL;DR
We’ve made significant progress in understanding LLMs, but there’s still much to uncover. While we’re good at the ‘what’ and making progress on the ‘how’, the ‘why’ remains a mystery. Let’s focus on addressing these unknowns and working together to advance the field.
Further reading: The State of LLMs: A Review of Recent Advances