Ai

Open LLM Landscape Shifts: New Commercial Models Arrive in 2024

Open LLM Landscape Shifts: New Commercial Models Arrive in 2024

The list of open large language models that businesses can actually use just expanded in a meaningful way. New entries, all announced in the first half of 2024, include Snowflake Arctic, Qwen1.5 110B, Fugaku-LLM, and Falcon 2. That's not just incremental progress. It's a signal that the open-source AI movement is increasingly targeting enterprise deployment, with bigger parameter counts and more permissive licenses.

The updated list still includes older models like Stability AI's StableLM-Alpha, FastChat-T5, and DLite from 2023. But the new additions clearly push the frontier forward.

What Changed

Snowflake Arctic is the heavy hitter. It's a 480-billion-parameter mixture-of-experts model with a 4,096-token context window, and it ships under Apache 2.0. That means you can use it for commercial products without paying a dime in licensing. Snowflake describes it as "the best LLM for enterprise AI," and the company is betting that open-weight models will eat into the proprietary AI market.

Then there's Qwen1.5 110B. Alibaba's Qwen team released this as the first 100B+ model in its Qwen1.5 series. It's got a 32,000-token context window, which is massive. But the license is not fully open. It's free for companies with under 100 million monthly active users, and you can't use Qwen outputs to train other LLMs unless they're Qwen derivatives. That's a real constraint for some businesses.

Fugaku-LLM is interesting for a different reason. It was trained on the Fugaku supercomputer in Japan, and it's a 13B model with a meager 2,048-token context. The license is custom and comes with usage restrictions. It's not the kind of thing you'd build a product on, but it shows how national supercomputing initiatives are entering the LLM race.

Falcon 2, from TII, is an 11B model with 8,192 tokens of context and an Apache 2.0 license with a mild acceptable use policy. The team claims it outperforms Meta's Llama 3. That's a bold statement, but the fact that it's Apache licensed matters a lot for startups.

Why It Matters

For enterprises, this is about more than just choice. The availability of open models under permissive licenses means you can deploy AI inside your own infrastructure, with your own security boundary, and keep control of your data. That's a huge deal for regulated industries like finance and healthcare.

But the licenses also create a trap. Qwen's restrictions might be fine for a small startup, but they could become a problem if you scale. The same goes for Fugaku-LLM's custom terms. It's essential to read the fine print before you commit.

One thing I find striking is how quickly context windows have become a differentiator. Qwen1.5 110B's 32k context is double what Snowflake Arctic offers. That could matter a lot for legal or code analysis use cases, where you need to feed in hundreds of pages of text.

The open LLM space is moving at breakneck speed. The models in this list were all released within about two months of each other. If you're building something commercially, the smart play is to keep an eye on this list, but also to model your architecture so you can switch between models as licenses or performance change. That's not just a technical choice. It's a business decision.

Official Source: https://github.com/eugeneyan/open-llms