Have you ever wondered what goes into building a language model like Qwen 2 from scratch? I recently took on the challenge and successfully implemented the 1.5B Qwen 2 model without any access to source code. This was a significant milestone for me, especially since there’s no open-source implementation of Qwen 2 available online.
What makes this build special is that it was based entirely on the Qwen 1 and Qwen 2 research papers. I also implemented the Qwen 2-1.5B architecture, with more sizes coming soon. One thing to note is that it doesn’t support Mixture of Experts (MoE) yet.
This project pushed my understanding of transformer architectures even further, and I’m excited to keep going. If you’re into large language models, model replication, or want to see how Qwen 2 works under the hood, this might interest you.
You can check out the source code on GitHub and Kaggle for a deeper dive into how it works.
I’m excited to see where this project takes me and how I can continue to improve my understanding of language models.