MythoTech AI News: 4/10/24

Mistral AI did it again, unexpectedly dropping a new open-source model by tweeting out a torrent link. Their last one (Mixtral 8x7b) proved to be incredibly powerful for its size thanks to the Mixture of Experts architecture that is rumored to be part of the secret sauce behind GPT-4. This setup uses multiple smaller models (the "experts") to work in parallel to produce better outputs faster, but at the cost of more compute to run. Check out this excellent blog post by Hugging Face for a much deeper and more technical explanation: https://huggingface.co/blog/moe

The main catch? This new model is HUGE by the current standards of open-source LLMs, with an 8x22b parameter sizing that makes it more than 3x bigger than Mixtral. VRAM is the very fast (and expensive) memory used by the Graphical Processing Units (GPUs) that are typically used to train and run these models. Given that Mixtral 8x7b (quantized) can only barely run on the 24gb VRAM used by the best consumer-grade GPU available today (the NVIDIA RTX 4090), we can assume this model will require at least 64gb of VRAM to run once it has been quantized which gets it comfortably out of reach for the average individual to run locally.

The good news? You can still try out this model today, as it's already available on Openrouter - a service that allows you to use a wide range of models in a single place. https://openrouter.ai/playground?models=mistralai/mixtral-8x22b

Since the model will presumably be offered as part of Mistral's API service, I will be adding it to my LLM API Factsheet once pricing and benchmarks are in!

It's been a very big week already for the open-source LLM community. Check out some of these other significant releases:

Code Gemma from Google: https://huggingface.co/collections/google/codegemma-release-66152ac7b683e2667abdee11

Command R+ from Cohere: https://huggingface.co/CohereForAI/c4ai-command-r-plus

StableLM 2 from Stability AI: https://huggingface.co/stabilityai/stablelm-2-12b

-----

This was originally posted on LinkedIn, and we invite you to join the conversation with us there!