OpenAI ships CSAM classifier with AI-generated content signal

OpenAI's new CSAM classifier scores images for human reviewers and flags likely AI-generated content, as AI-related reports jumped 8,400% from 2023 to 2025.
By Nathan Brooks
3 min readAt a glance
- AI-related CSAM reports rose more than 8,400 percent globally from 2023 to 2025, per Childlight.
- NCMEC received over 400,000 AI-related CSAM reports last year, up from roughly 4,700 in 2023.
- OpenAI says its classifier is the first broadly available tool with a signal for AI-generated CSAM.
- The tool flags nearly half as many images while finding roughly the same number of abusive images as OpenAI's existing tool.
- ROOST president Camille François called the model card the first in its field.
AI-related reports of child sexual abuse material jumped more than 8,400 percent globally between 2023 and 2025 — and on Monday, OpenAI answered with a new detection tool built for exactly that surge. The company released a CSAM classifier that scores images for human reviewers and adds a signal indicating whether an image may have been generated with AI. OpenAI says it's the first broadly available tool of its kind to include that AI-specific flag.
Why the tool arrives now
The numbers behind the launch are stark. A new analysis by Childlight, the Global Child Safety Institute at the University of Edinburgh, found AI-related CSAM reports grew by more than 8,400 percent from 2023 to 2025. The National Center for Missing & Exploited Children (NCMEC) received more than 400,000 of these reports last year. In 2023, that figure sat at roughly 4,700.
That's an 85-fold increase in two years, and it's what pushed detection from a niche capability into shared infrastructure.
How the classifier actually works
The tool scans images for visual signs of abuse and assigns each one a score. Human reviewers use those scores to decide which images warrant attention first. It's a triage system, not an automated judge — people stay in the loop.
Access is restricted. Only organizations required or authorized to detect CSAM can use it, including certain developers building on OpenAI's API. OpenAI also made the classifier interoperable with Coop, an open source detection-and-reporting tool from the nonprofit ROOST. Coop lets users flag content, review it, and report it to NCMEC.
In its announcement, OpenAI framed the release as both a corporate duty and an industry obligation: "Individual companies like OpenAI have a responsibility to prevent our technology from being exploited by predators to harm children, and we also must work together with the broader industry, child safety organizations and law enforcement to advance our collective efforts to protect children online."
What does the model card say?
Alongside the classifier, OpenAI published a model card — the documentation format it routinely uses to explain how products like ChatGPT were refined and tested. Camille François, president of ROOST, told Mashable it's the first model card in the child-safety field.
"OpenAI is setting an important precedent for transparency in child safety," François said in an email. "Builders need to know how well detection tools work and where they fall short. Publishing that evidence helps teams build better safeguards and gives the whole field a clearer view of where to focus research."
The card includes early testing results. OpenAI estimates the classifier could flag nearly half as many images as its existing tool while finding roughly the equivalent number of abusive images. In plain terms: fewer false positives eating up reviewer time, with detection quality held steady.
What it means for trust-and-safety teams
For practitioners, the practical gain is review capacity. Halving the volume of flagged images lets the same headcount focus on the material most likely to be real abuse. The AI-generation signal also matters for escalation, since AI-generated CSAM often falls into different legal and reporting channels than photographic material.
The transparency angle shouldn't be overlooked either. François's point about publishing evidence cuts both ways — it lets outside teams benchmark the classifier rather than take OpenAI's claims on faith. That's rare in a field where detection tools usually ship as black boxes.
OpenAI hasn't said what performance gains, if any, future versions will target, but the model card gives the industry a baseline to measure against as AI-generated CSAM volumes keep climbing.
Source: mashable.com
Nathan Brooks
Show full bio
Staff writer covering companies, deals and strategy in Social platform news and policy at Reach Brief.
Related articles
AI Child Abuse Reports Jump 8,400% in Two Years, Data Shows
Five AI Detectors Flag Viral 'Saluting Veteran' Parade Video as Fake
OECD Flags Instagram Algorithm Over Rapid Antisemitic Content Routing
TikTok Rolls Out New AI Literacy Measures for Users
Meta Rolls Out Instagram AI Age Detection Globally to Push Teens Into Teen Accounts