How or Where to check Performance of Open Source LLM?

To check the performance comparison for open-source large language models (LLMs), you can refer to several resources and platforms. Here’s a summary of some key sources:

1. Hugging Face’s The Big Benchmarks Collection: Hugging Face provides an Open LLM Leaderboard which tracks, ranks, and evaluates open LLMs and chatbots. They offer various leaderboards such as the Open LLM Leaderboard, MTEB Leaderboard, and Chatbot Arena Leaderboard. These leaderboards compare models based on different benchmarks and metrics like accuracy, latency, throughput, and memory performance​​.
2. Shakudo’s Comparison of Open-Source LLMs: Shakudo provides detailed comparisons of several open-source LLMs such as the MPT series, FastChat-T5, and OpenLLaMA. They discuss each model’s architecture, training computational requirements, use cases, applications, and performance improvements. You can learn more about how to use these models for inference and integration on their respective pages on Hugging Face​​.
3. Plain English’s Detailed Comparison: Plain English offers insights into various models like StableLM, Dolly, and GPT4All, including how to run these models using Hugging Face’s transformers library. They also discuss the model’s licensing, datasets used for training, and integration into applications​​.
4. Sapling’s LLM Index: Sapling provides an index of various LLMs, including BLOOM, BLOOMChat, Cerebras-GPT, Dolly, Falcon, FastChat, FLAN-T5, etc., with links to more detailed information and comparison data​​.
5. Deci’s Top 10 List of Open-Source LLMs: Deci’s article outlines key details of various open-source LLMs like LLaMA 2, Alpaca, and Vicuna. It covers their training data, architecture, applications, and specific features that distinguish each model. The article also discusses the models’ intended use, whether for research or commercial applications​​.

You can visit these platforms and resources to get detailed comparisons and performance metrics of various open-source LLMs to understand their capabilities and choose the one that best fits your needs.