Billionaire multi-company founder Elon Musk’s startup xAI today released its first large language model (LLM), Grok, as open source.
The move, which Musk had previously proclaimed would happen this week, now enables any other entrepreneur, programmer, company, or individual to take Grok’s weights—the strength of connections between the model’s artificial “neurons,” or software modules that allow the model to make decisions, accept inputs, and provide outputs in the form of text—and other associated documentation and use a copy of the model for whatever they’d like, including commercial applications.
“We are releasing the base model weights and network architecture of Grok-1, our large language model,” the company announced in a blog post. “Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI.”
Grok’s code can be downloaded via a torrent link or from its Github page.
xAI open sources Grok: Understanding its impact
The model’s weights and biases are referred to as parameters; the more parameters there are, the more advanced, complex, and performant the model is. Grok, with 314 billion parameters, outperforms open-source competitors like Meta’s Llama 2 (70 billion parameters) and Mistral 8x7B (12 billion parameters).
Grok was open-sourced under the Apache License 2.0, which allows for commercial use, modifications, and distribution, but it cannot be trademarked, and users receive no liability or warranty. In addition, they must reproduce the original license and copyright notice, as well as state the changes they made.
Grok’s architecture, which was developed in October 2023 using a custom training stack based on JAX and Rust, incorporates novel neural network design approaches. The model uses 25% of its weights for a given token, which increases its efficiency and effectiveness.
Grok was first released as a proprietary or “closed source” model in November 2023, and it was previously only available on Musk’s separate but related social network X (formerly Twitter), specifically through the X Premium+ paid subscription service, which costs $16 per month or $168 per year.
However, Grok’s release does not include the entire corpus of training data. This does not really matter for using the model because it has already been trained, but it does not allow users to see what it learned from — presumably user text posts on X. (The xAI blog post describes it vaguely as “Base model trained on a large amount of text data, not fine-tuned for any particular task.”)
It also lacks access to the real-time information available on X, which Musk had previously touted as a key advantage of Grok over other LLMs. Users will still need to subscribe to the paid version of X.