Have you ever wondered why autoregressive decoders dominate the landscape of language models, despite their limitations? One major drawback is that they’re prone to hallucinations, which are baked into their probabilistic framework. Moreover, they face an inherent trade-off between early, middle, and late tokens, which affects the quality of their output.
On the other hand, Diffusion-Encoder type models seem to offer a more promising approach. By allowing all tokens to see each other, they eliminate the ‘goldilocks’ problem and can decode entire sequences at once, making them computationally efficient. Plus, diffusion models focus on finding high-probability manifolds, which should reduce the likelihood of hallucinations.
So, why aren’t Diffusion-Encoder LLMs more popular? One challenge lies in adapting diffusion models from continuous image domains to discrete text tokens. However, we already use embeddings to make tokens continuous, so it’s unclear why we can’t apply diffusion in embedding space.
Despite these challenges, Google has developed a diffusion LLM, and it’s time for open-source alternatives to emerge. Additionally, there’s room for innovation in attention mechanisms, such as using convolutions and the Fast Fourier Transform to operate over full sequences in O(N log N) time.
The potential of Diffusion-Encoder LLMs is vast, and it’s exciting to think about the possibilities they could unlock.