YouGov has tested its AI-generated survey respondents against the real people they were modeled on, and it is now pushing the research industry to adopt shared rules for checking simulations. The international research and data analytics group published the white paper on Tuesday, describing what it calls one of the most transparent tests yet of one-to-one human-to-digital-twin simulation.

Most published validation work compares aggregate results against public datasets. YouGov took a different route. Working with OriginalVoices, an AI platform that specializes in simulation, the company built digital twins, which it calls “E-twins,” of 1,008 members of its proprietary panel. Each member was verified with a government ID. YouGov then put the same new questions to the E-twin and to the real panelist behind it.

The panelists answered the follow-up questions quickly. About 69% responded within 48 hours, which gave the company a direct way to check how accurate each twin was. YouGov says that in day-to-day operations, this kind of validation can be done in as little as an hour.

How accurate are YouGov's AI twins?

The headline number concerns training data. Twins trained on panelists’ ratings of more than 4,000 brands and other entities matched their human counterpart to within one point on a five-point scale 90% of the time. Without that rich data, and relying on profiling data alone, the match rate was 78%.

Can AI twins answer questions they were never trained on?

The twins also performed well on material they had never seen. YouGov did not include fiction books in the training data. When asked about them, the E-twins still came within one point of their human 90% of the time.

The company also found that the twins’ own confidence ratings tracked with accuracy. OriginalVoices assigns each open-ended answer a confidence level of High, Medium or Low. Panelists said the answer described their own views “very” or “somewhat” well for 81% of High-confidence answers, 72% of Medium and 67% of Low. Researchers can use those levels to decide which individual twin responses to trust. Overall, 75% of the time, panelists judged their E-twin’s open-ended answers as describing their own views.

Stereotyping was another concern. Adding more contextual training data reduced the twins’ reliance on demographics. YouGov reported no pattern of biased performance by age, gender, race or education.

The white paper is also a critique of how rivals validate their products. YouGov argues that much of the evidence simulators have offered so far relies on publicly available datasets. Those questions may already be in the training data of the underlying language models, so a twin can seem to know an answer it has simply absorbed. Because respondents usually cannot be recontacted, the results are hard to verify. Aggregate comparisons can also hide twins that flatten their answers toward the average or fall back on stereotypes.

What verification standards does YouGov propose for AI simulation?

YouGov wants the industry to develop shared standards of verification and validation for defined use cases. It challenges every simulator to validate twins against their individual human, test them on topics they were not trained on, publish their methods, show their misses and predict real-world outcomes in public.

Stephan Shakespeare, CEO and co-founder of YouGov, framed the issue as one of trust. “The research industry is being transformed by AI in many ways but especially when it comes to simulation,” he said. “Whether this is a dangerous development or a beneficial advance depends on whether clients can know if its outputs are reliable. I challenge everybody in the industry to work together to establish standards for verifying and validating simulated data.” He added: “If the industry wants simulation to be trusted by clients and the public, it needs to earn it and this can only be achieved through widely agreed standards.”

The paper also lists what it did not measure. YouGov did not establish a ceiling, meaning how often a person gives the same answer to the same question when asked again. That ceiling matters because a twin cannot reasonably beat the person it copies. As an illustration, the paper notes that if panelists repeated their own rating 70% of the time, a twin’s 58% exact-match rate would represent 83% of that ceiling. YouGov says it will test the ceiling in the next iteration of its research.

Dieser Artikel wurde mit Unterstützung von KI verfasst.
News Factory APP - agentische News für besseres SEO & AEO.