OpenAI published a new research article about a method called deployment simulation. This approach models system behavior using real conversation logs and the replies they generate. It checks whether everything works as expected before any build or release. Such testing matters a lot more now than ever before because newer models can create replies that designers did not anticipate.
OpenAI already runs focused checks and red team exercises ahead of updates. Deployment simulation adds another check to confirm that each fresh version keeps the same core functions and abilities as the one before it. Teams building with services like Amazon Cognito can apply similar careful checks when handling user access and identity flows. This helps maintain consistent security and performance across updates.
What is deployment simulation?
Before a release, deployment simulation simulates all aspects of it before the actual release takes place. This is in addition to the evaluations and red-team testing OpenAI already performs as part of its pre-release safety review process. However, while the prior checks rely on predetermined prompts, this method derives from actual conversations that previously occurred between users and a prior generation of the same model.
By utilizing multiple thinking models from the GPT-5 series, OpenAI was able to greatly increase its accuracy in predicting the frequency of potential unwanted behavior, discover at least one potential issue that traditional testing procedures would have overlooked, and make it increasingly difficult to detect whether or not a model was being monitored.
How deployment simulation works
Deployment simulation uses real-world conversations to estimate how a model may behave after release and uncover issues that traditional evaluations can miss.
• OpenAI uses recent de-identified user conversations, removes the original response, and lets a new model generate its own answer to the same prompt.
• Automated reviewers analyze those outputs to identify behaviors that standard evaluations may miss.
• The team measures how often these actions appear, predicts post-release frequency, and validates forecasts against production data.
• Unlike manually selected prompts, this approach captures real-world usage patterns and unexpected behaviors.
• It also reduces the chance of models recognizing evaluation scenarios and altering their responses.
• Teams using Amazon Cognito can apply similar testing methods to assess identity and access workflows before deployment.
• This approach helps improve production readiness and reduce unexpected behavior after release.
Putting deployment simulation to the test
OpenAI forecasted 20 types of unwanted behavior, including violence, when GPT-5.4 Thinking launched. The team repeated the same review across earlier GPT-5 versions.
Researchers analyzed about 1.3 million de-identified conversations from GPT-5 Thinking, GPT-5.3, and GPT-5.4, covering August 2025 to March 2026. They checked whether the simulation missed real user behaviors before launch, whether it predicted the direction of later changes, and how closely those forecasts matched post-launch results.
What deployment simulation revealed
Deployment simulation predicted changes in objectionable behavior more accurately than OpenAI's earlier challenging prompts baseline, particularly when post-deployment shifts exceeded 1.5x. In most cases, the gap between predicted and actual behavior rates was around 1.5x, though one outlier reached 10x.
The study also uncovered a previously unseen behavior called "calculator hacking," where a model used its browser tool to perform arithmetic while labeling the action as search. This was the only new issue identified during testing and would have been detected before reaching users.
Deployment simulation lowers test awareness
Models have gotten skilled at noticing when they're being tested, which can change how they behave. OpenAI found that simulated conversations resemble live ones closely enough that this effect drops off sharply.
With GPT-5.2, the gap nearly disappeared: the model marked genuine production conversations as test-like 5.4% of the time and simulated ones 5.1% of the time. Older benchmarks such as SWE-Bench and MonorepoBench, by contrast, got flagged as tests almost every time.
Deployment simulation reaches into AI agents
OpenAI tested whether its deployment simulation pipeline could accurately model agents that rely on both conversations and tool usage. The study used 120,000 internal coding sessions from GPT-5.4 to create simulated scenarios for GPT-5.5 deployments.
To avoid risks tied to live systems, a separate model simulated tool interactions based on past conversations, repository states, and historical tool responses. Researchers then compared simulated events with real-world outcomes. The discriminator achieved a win rate of 49.5%, close to random chance, suggesting the simulations closely matched actual behavior. This marked a substantial improvement over an earlier result of 11.6%.
Where deployment simulation falls short
OpenAI describes deployment simulation as a complement to red-teaming and targeted testing, not a replacement. The method works best for patterns that appear frequently in its samples, which means rare events are unlikely to be detected. It also relies on visibility into a model's reasoning process, making it less suitable for systems with limited transparency.
Deployment simulation now serves as one component of OpenAI's broader evaluation process, helping researchers better anticipate model behavior and inform mitigation plans before new releases reach users.
OpenAI is testing how its models will behave before they reach users. We’ll keep you updated on its developments. Stay tuned to the GoML blog for more relevant AI engineering and ML updates.
