What is Artificial Analysis Intelligence Index version 5.0?

Artificial Analysis has released version 5.0 of its Intelligence Index, the composite score the firm uses to rank and compare language models. The index now rests on nine evaluations split evenly across four categories, each carrying 25 percent of the total weight: Knowledge Work, Coding, Science and General.

How are the Intelligence Index categories weighted?

The Knowledge Work category is led by AA-Briefcase v1.1, a benchmark of multi-week knowledge work projects that carries 15 percent of the score. AutomationBench-AA and GDP.pdf each account for another 5 percent. Coding combines FrontierCode 2.0, which asks agents to make changes to real open-source repositories and is weighted at 15 percent, with Terminal-Bench 4.0 at 10 percent. Science pairs Terminal-Bench-Science 0.1 at 15 percent with the hard subset of CritPt v2 at 10 percent. The General category holds AA-Omniscience v1.1, weighted at 15 percent and split between answer accuracy at 10 percent and a non-hallucination rate at 5 percent, along with HLE-Diamond at 10 percent.

Which benchmarks were removed from the Intelligence Index?

Several benchmarks changed places in the process. Terminal-Bench-Science, FrontierCode 2.0 and HLE-Diamond are new to the index. The GDP.pdf weight dropped from 10 percent to 5 percent, and the CritPt benchmark was replaced by its v2 hard subset, which keeps the same 10 percent weight. Four evaluations were removed from the index entirely: GDPval-AA, SciCode, AA-LCR and Humanity's Last Exam. The company's version history lists the revision as current from October 2026 onward.

HLE-Diamond is a refined, 1,000-question version of Humanity's Last Exam, built by the Center for AI Safety and Scale AI. The questions are split evenly between 500 reasoning items and 500 knowledge items. Artificial Analysis evaluates all 1,000 questions without tools, and it grades answers using an LLM judge that follows the official HLE evaluation scripts.

AA-Omniscience v1.1 is the company's knowledge and hallucination test. It now contains 3,000 questions covering 42 topics, down from 6,000 in the original dataset. The company removed saturated questions, dropped 600 questions that had been published in the v1 public set, added new items and corrected ambiguous ones. Because of those changes, the firm says scores are not directly comparable with v1 results. The scoring system awards points for correct answers, subtracts points for hallucinated responses and treats abstentions as neutral, so a model that declines to answer does better than one that guesses wrong.

The Intelligence Index is primarily a text-based, English-language suite. Artificial Analysis tests image inputs, speech inputs and multilingual performance separately, and those results do not feed into the index score. The company also notes that the index has limits. In its methodology document, it states that like all evaluation metrics, the index may not apply directly to every use case. Even so, the firm says it is confident the index is a more useful comparison between language models than any other metric available today.

Some components use pairwise judging and Elo ratings, which the company fits with a Crowd-BT model. For AA-Briefcase, a model's combined Elo is frozen at the time it is added to the index and then normalized to a 0-to-1 range using a fixed formula. The Elo scale is anchored to GPT-5.5 (medium) at 1000. Artificial Analysis says it may update those reference parameters as models improve, in order to keep the index able to tell strong models apart.

Cet article a été rédigé avec l'assistance de l'IA.
News Factory APP - actualités agentiques pour booster votre SEO et AEO.