Who can access OpenAI's child abuse detection classifier?
OpenAI has released a new image classifier that can detect potential child sexual abuse material, including material that may be AI-generated, and is offering it free to vetted organizations through its API. The company said the tool is the first CSAM classifier with broad availability that includes a specific signal for whether the material may have been made with AI.
The classifier looks for visual signs of abuse, which helps teams find images they haven't seen before. It assigns scores that help reviewers decide which images to look at first. OpenAI said its teams built and tested the tool for real-world deployment with assistance from the National Center for Missing and Exploited Children, or NCMEC, a nonprofit that runs the CyberTipline, a centralized reporting hub for online exploitation of children.
How the classifier performed in testing
How accurate is the classifier in testing?
In an evaluation of user-uploaded images, OpenAI compared the new classifier with its existing protections. The company estimated the new tool would flag 48% fewer images while finding roughly the same number of abusive images. Images flagged by the new classifier were about twice as likely to contain abuse.
Early access partners also tested the system. Fal reported that human reviewers confirmed 72% of the classifier's high-confidence flags as novel CSAM in testing on production images. Invideo AI said one in four flagged cases led to a ban. In SafetyKit's evaluations involving drawings or animations that sexualize children, the classifier detected 90% of cases.
OpenAI is making the tool available through the Tech Coalition and ROOST, which will help share access. The company also said it supports tools built by others, such as Thorn's Safer CSAM detection system, and has begun participating in the Tech Coalition's Lantern program, where members share signals about online child sexual exploitation and abuse.
Warnings and a 24-hour pause in ChatGPT
Does ChatGPT warn users who search for abusive material?
OpenAI already bans CSAM, trains its models to refuse requests for it, and takes action against accounts that break its rules. The company is now adding product warnings in ChatGPT that appear when a user tries to find abusive material. The warning explains the consequences and points people toward help to stop the behavior. It also states that sexualizing children causes harm, including when the material is AI-generated.
Alongside the warning, OpenAI will remove the flagged conversation and pause access to ChatGPT for 24 hours. Specialist support will remain available. Repeated violations may bring longer restrictions, account closure or reports to authorities.
The company said it developed the approach with input from child abuse prevention groups and academic advisers. They include Protect Children, the Lucy Faithfull Foundation, Dr. Caoilte Ó Ciardha of the University of Kent, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages.
Guidance for developers and investigators
OpenAI has also published CSAM guidance for developers. Written with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation, the document covers clear rules, ways for users to report concerns, detection tools and processes for reviewing and reporting suspected abuse. It explains how safety identifiers can link misuse to individual users and how to support staff who review sensitive content. It is designed for organizations at different stages of building their safety systems.
Separately, OpenAI worked with Anthropic and Google to commission guidance on responding to disclosures of child sexual exploitation and abuse in AI systems. The Tech Coalition recently published it. OpenAI said the guidance has already led to more developers accessing child safety tools and registering to report CSAM to NCMEC.
For law enforcement, OpenAI is rolling out tools and training for child-safety investigative units, with access managed by program administrators and dependent on agency authorization. The first tool will help investigators review lengthy chat logs and legal process returns, identify potential connections and create timelines, with findings linked to source material. The initial training will be an overview and demonstration, followed by a field guide. OpenAI said future training sessions will be proposed at regional, national and international conferences.
OpenAI said the work is meant to prevent harm to children and strengthen the response when abuse occurs. The company acknowledged that stopping online predators requires coordination across many organizations, and it said it will keep improving its safeguards while working with child protection groups, developers, researchers and investigators.
This article was written with the assistance of AI.
News Factory APP - agentic news to boost your SEO & AEO.
