Ask an AI assistant for a statistic and you will get a number, a confident tone and a link. Whether that link leads to the research behind the figure is another matter. A new test of five popular assistants suggests it usually does not.

How often do AI assistants cite the original study behind a statistic?

Trakkr Research asked ChatGPT, Perplexity, Gemini, Claude and Google’s AI Overviews 200 questions that call for a number. It then followed the citations for a random sample of 1,285 statistics, clicking through each cited page to its source and continuing up to five pages deep. Where a page named a source without linking to it, researchers searched for it. Only 40% of the trails reached a study that said the same thing.

Another 8% reached the study, but the number had changed along the way. The remaining trails went nowhere useful. In 40% of cases the trail stopped at a source that could not be checked because it was locked, gone or silent on the figure. In 11% of cases, no study existed anywhere on the trail. Every page repeated the number and credited no one.

ChatGPT fared best. It sourced 57% of its statistics and quoted the fewest, averaging 1.3 per answer. Claude quoted the most, at 6.8 per answer, and was sourced 38% of the time. Perplexity came in at 46%, AI Overviews at 39%, and Gemini at 38%. Researchers said the four assistants other than ChatGPT were too close to rank.

Which sources give the most reliable AI-cited statistics?

The first page a trail lands on matters a great deal. AI assistants most often cited company blogs, which made up 54% of first citations, and only 37% of those trails reached a matching study. Research firms and government pages did far better, reaching a study 62% and 61% of the time. Only 21% of the pages cited for a number were original research. The cited page did usually contain the figure, though. It had the number 82% of the time.

Numbers that drift

Reaching the original study did not always mean the figure held. Of the statistics that did reach a study, 17% had changed. The digits themselves moved in roughly one case in four. More often, the context shifted: what was measured, the year, the source or who was asked.

One example involved a 2004 study in which 15 British women judged health websites. Its finding that 94% of their comments about rejecting a site concerned design was later retold as a claim that 94% of first impressions of any website are design-related. Google AI Overviews presented it as fact.

Why do AI assistants give different numbers for the same question?

The assistants also often disagreed with one another. On 28 of 71 questions where all five gave a percentage, the answers included four or five different figures. Asked how many marketers use marketing automation tools, the five gave answers ranging from 49% to 96%. ChatGPT said the figure varied by survey and by what counted as marketing automation.

Figures every assistant repeats

Agreement did not signal reliability. The assistants gave the same figure on 10 questions, but the researchers traced only a handful of those and found just two sourced. Statistics quoted by three or more assistants fared little better, with 11 of 29 sourced.

Four statistics were quoted by all five assistants, and researchers could not find a study behind any of them. The most widely repeated was the claim that 44% of salespeople give up after one follow-up attempt. The trail ended at a freelance writer’s sample article, online by 2017, that cited no research. Another was the claim that 90% of social media users follow at least one brand, which was stated without any source. A third said 75% of searchers never scroll past the first page of results, often credited to an old marketing roundup that no longer contains it. The fourth said it costs five to 25 times more to win a customer than to keep one. A 2014 Harvard Business Review article gave that range “depending on which study you believe,” and named none.

Trakkr’s advice to anyone using AI-generated numbers is to treat them as leads, not sources. Follow the citation further, check who was asked, how many people responded and when, and do not treat five matching answers as five independent confirmations. The company also advises publishers to put the sample, date and a link to the original source next to any figure they publish, so that the next page in the chain can be traced back to them.

Questo articolo è stato scritto con l'assistenza dell'IA.
News Factory APP - notizie agentiche per potenziare il tuo SEO e AEO.