OpenAI has rolled out a package of tools aimed at combating child sexual abuse material, or CSAM, including a new image classifier that can detect abusive images, including ones generated by AI. The company also is adding warnings to ChatGPT and expanding its resources for developers and police. The move comes as concerns grow about how AI tools can be misused.
What does OpenAI's new image classifier do?
The classifier is available to select companies through OpenAI’s API at no cost. It looks for visual signs of abuse in images it has never seen before and assigns a score that helps reviewers decide how to handle a file.
OpenAI developed the classifier with the National Center for Missing and Exploited Children, or NCMEC, and is working with Tech Coalition and ROOST to share access to the tool. In early testing with partners, the company said the classifier flagged 48% fewer images while still finding the same number of abusive images, which cuts down on the work for human reviewers. Images flagged by the tool were also twice as likely to be abusive.
What happens when ChatGPT users request child sexual content?
ChatGPT will now show a warning to users who attempt prompts seeking content that sexualizes children. The alert points them toward specialized support intended to stop harmful behavior. Users who try it also face a 24-hour account suspension as a warning. The app already blocks CSAM content and requests related to it, the company said, and repeated offenses could lead to the account being permanently removed.
OpenAI said it shaped the messaging and guidelines with several organizations and researchers. They include Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan at the University of Tasmania, who independently evaluated a trial of the deterrent messages.
The company is also publishing guidelines for developers who use OpenAI tools, meant to help them prevent, detect and report CSAM. OpenAI said early testing showed the guidelines helped a number of developers adopt child safety tooling and register to report CSAM to NCMEC.
How is OpenAI supporting law enforcement against CSAM?
For law enforcement, OpenAI is offering AI-powered features such as dedicated plugins, investigative tools and complex data analysis. Access will be managed by program administrators. The company is also providing training for investigators, along with a field guide on how to use AI.
The announcement arrives against a backdrop of scrutiny. OpenAI has faced several lawsuits accusing its technology of harming users, especially children. The safety-focused nonprofit Common Sense Media has labeled OpenAI’s technology an “unacceptable risk” for teens. Separately, other AI chatbots, Grok in particular, have been used to create nonconsensual sexualized imagery, including images of children.
OpenAI said it has made several changes over the past 12 months. They include releasing a safety blueprint, updating its content guidelines to bar inappropriate content, retuning its model to prevent it from generating harmful content, and releasing a special version of ChatGPT for teens.
Este artigo foi escrito com a assistência de IA.
News Factory APP - notícias agênticas para impulsionar seu SEO e AEO.