
OpenAI’s chief scientist, Jakub Pachocki, wants every frontier AI lab to track one shared number: perplexity. In a new post, he publishes OpenAI’s own internal perplexity scores and proposes an AI safety standard built on agreed limits that no developer would cross until safety tools catch up. It is a rare move, since it hands competitors a clearer view of where OpenAI stands.
Inside the Proposed AI Safety Standard
Perplexity measures how well a model predicts the next token in text it has never seen. Lower is better, and OpenAI usually tracks the log of it, also called cross-entropy loss. Pachocki argues this single score is the best way to follow the raw “intelligence” a model gains during pretraining.
His proposal has three parts. Labs would report perplexity openly, agree on shared norms for testing and red-teaming very capable models, and commit to common limits on development. Those limits would only be lifted once the industry reaches a concrete safety milestone, such as a successor to today’s chain-of-thought monitors.
Why OpenAI Shared Its Own Numbers
Pachocki admits that publishing internal measurements took some thought because it helps rivals. He says that concern is outweighed by the urgency of building shared safety measures across the industry. OpenAI frames the release as a first step toward a common scientific basis for pacing AI progress.
OpenAI’s Perplexity Scores by Model
The numbers come from OpenAI’s main held-out dataset, which is drawn from its internal codebase. That means older models like GPT-2 and GPT-3 look weaker on it than they would on general text. Here is how the reported scores line up with the abilities OpenAI says showed up at each step.
| Model or threshold | Log-perplexity | Capability OpenAI links to it |
|---|---|---|
| GPT-3.5 | About 0.99 | Fluent conversation, the basis for the original ChatGPT |
| GPT-4 | About 0.91 | Useful chain-of-thought reasoning |
| Recent models | Below 0.8 | Major gains in cybersecurity and mathematics |
| Recent math model | About 0.76 | What OpenAI calls groundbreaking results on open math problems |
| Next window | About 0.68 to 0.7 | Further transformative abilities expected, not yet known |
OpenAI’s chart also shows a steep drop in loss during 2026. The company credits that mostly to long-term research and compute investment, with a smaller share from AI automation. Still, it says its newest models already received meaningful design contributions from AI itself.
Why AI Helping Build AI Matters
The biggest worry in the post is speed. Pachocki says AI doing algorithmic research on pretraining could cause a sudden jump, much like the burst of math results OpenAI saw once its models crossed a certain level. He suggests a similar wave of pretraining advances could arrive as soon as 2027.
He also expects safety checks to get harder. As models improve, he predicts current alignment audits will weaken and models will become more aware of when they are being tested. In his view, alignment and monitoring research probably won’t keep pace on its own, which is why he wants agreed limits in place.
The Weak Spots in a Perplexity Safety Standard
OpenAI is open about the problems with using perplexity this way. The post lists several challenges that any shared standard would need to solve:
- Labs use different test datasets and training data, so scores are hard to compare directly.
- Perplexity does not map to any real task people care about.
- It misses gains made after pretraining, such as chain-of-thought reasoning.
- No one knows the lowest possible score, and small gains near that floor may mean big leaps in ability.
Even so, Pachocki argues perplexity has held up for years while benchmarks keep getting saturated. Tests used for GPT-3 were mostly maxed out by GPT-4, but pretraining loss stays meaningful across generations.
What This Means for You
For everyday users of tools like ChatGPT and rivals, nothing changes today. This is a proposal, not a product update or a binding rule. But it signals how one of the biggest labs thinks about the pace of progress and where it sees risk.
- Expect more debate over shared limits, alongside efforts like binding UK regulation and the FTC safety probe.
- Watch whether other labs publish their own scores, much like the detail in the Gemini 4 Argon safety report.
- If you work in security or research, OpenAI’s claim that big jumps show up at specific loss levels is worth tracking.
You can follow how this plays out in our daily AI news coverage, including related moves like Anthropic’s safety bill draft.
Sources
Primary source: OpenAI’s post on perplexity and intelligence by chief scientist Jakub Pachocki.
Conclusion
OpenAI is betting that a single, shared metric could give the AI industry a common way to measure progress and agree on when to slow down. By publishing its own perplexity scores, it has made the first move and invited rivals to follow. The idea still faces real hurdles, from mismatched datasets to the gap between loss and real-world skills. Whether other labs sign on will decide if this becomes a true standard or stays a thought experiment. Either way, it is one of the clearest public signals yet that OpenAI expects AI progress to speed up.


