A website called ArtificialAnalysis benchmarks the intelligence of various LLM models and publishes a headline Intelligence Index. The index is calculated as the mean output of a curated selection of benchmarks run on each model. However, their plot of Intelligence vs. cost has several issues, including a logarithmic scale and the use of official pricing from model developers.
The logarithmic scale makes it difficult to appreciate the immensity of the price difference between cheap and expensive models. Additionally, the official pricing may not reflect the true cost of running the models, especially for open-weights models that can be rented from third-party API providers at a lower cost.
A revised plot, created using the same intelligence index scores from ArtificialAnalysis, but with a linear scale and adjusted pricing, shows a more accurate representation of the cost-effectiveness of each model. The plot highlights the diminishing returns of increased intelligence, with top-tier models being extremely expensive and only marginally more capable than cheaper alternatives.
Some models, marked with a symbol, can be run locally on consumer hardware, with their cost calculated based on the electricity required to run them. These models are extremely cheap, with costs ranging from $0.015 to $0.023 per task, making them accessible to a wide range of users.
The revised plot also shows that the cost of running models locally can be substantial, especially for larger models that require more powerful hardware. However, even these models can be more cost-effective than running them in a datacenter, especially for users who already own the necessary hardware.
Overall, the revised plot provides a more accurate and informative representation of the cost-effectiveness of LLM models, allowing users to make more informed decisions about which models to use and how to run them.
Cet article a été rédigé avec l'assistance de l'IA.
News Factory APP - actualités agentiques pour booster votre SEO et AEO.