GPT-5.4 Thinking System Card

GPT-5.4 Thinking is the latest reasoning model in the GPT-5 series, and explained in our blog. The comprehensive safety mitigation approach for this model is similar to previous models in this series, but 5.4 Thinking is the first general purpose model to have implemented mitigations for High capability in Cybersecurity. The approach to cyber safety builds on the latest approaches implemented for GPT-5.3 Codex, in ChatGPT and the API.

In this card we also refer to GPT-5.4 Thinking as gpt-5.4-thinking. Note that there is not a model named GPT-5.3 Thinking, so the main model to baseline against is GPT-5.2 Thinking.

Like OpenAI’s other models, GPT-5.4 Thinking was trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide or generate. Our data processing pipeline includes rigorous filtering to maintain data quality and mitigate potential risks. We use advanced data filtering processes to reduce personal information from training data. We also employ safety classifiers to help prevent or reduce the use of harmful or sensitive content, including explicit materials such as sexual content involving a minor.

OpenAI reasoning models are trained to reason through reinforcement learning. These models are trained to think before they answer: they can produce a long internal chain of thought before responding to the user. Through training, these models learn to refine their thinking process, try different strategies, and recognize their mistakes. Reasoning allows these models to follow specific guidelines and model policies we’ve set, helping them act in line with our safety expectations. This means they provide more helpful answers and better resist attempts to bypass safety rules.

Note that comparison values from previously-launched models are from the latest versions of those models, so may vary slightly from values published at launch for those models.¹

We conducted benchmark evaluations across disallowed content categories. We report here on our Production Benchmarks, an evaluation set with conversations representative of challenging examples from production data. As we noted in previous system cards, we introduced these Production Benchmarks to help us measure continuing progress given that our earlier Standard evaluations for these categories had become relatively saturated.

These evaluations were deliberately created to be difficult. They were built around cases in which our existing models were not yet giving ideal responses, and this is reflected in the scores below. Error rates are not representative of average production traffic. The primary metric is not_unsafe, checking that the model did not produce output that is disallowed under the relevant OpenAI policy.

Values from previously-launched models are from the latest versions of those models, and evals are subject to some variation. Values may vary slightly from values published at launch for those models.

gpt-5.4-thinking generally performs on par with gpt-5.2-thinking with statistically significant improvements on illicit non-violent activity and self-harm evals.

Ahead of the GPT-5.3 Instant launch, we implemented dynamic multi-turn evaluations for mental health, emotional reliance, and self-harm that simulate extended conversations across these domains. Rather than assessing a single response within a fixed dialogue, these evaluations allow conversations to evolve in response to the model’s outputs, creating varied trajectories during testing that better reflect real user interactions. This approach helps identify potential issues that may only emerge over the course of long exchanges and provides an even more rigorous test than prior static multi-turn methods. By utilizing realistic, yet adversarial user simulations, these evaluations have enabled continued improvements in safety performance, particularly in areas where earlier evaluation frameworks had reached saturation.

Our standard evaluations measure whether the final model response violates our policies. In these dynamic conversations, we instead evaluate whether any assistant response violates policy and report the percentage of policy-compliant responses. The metric used is not_unsafe, representing the share of assistant messages that do not violate safety policies.

Across all dynamic mental health evaluations, gpt-5.4-thinking outperforms previous models.

In addition to our standard disallowed content evaluations, we also piloted an additional check as part of this deployment, estimating rates of safety-relevant model behaviors on a production-like distribution of deidentified user traffic (in compliance with OpenAI’s privacy policy). This pilot was based on our recently published research.

Before release, we used deidentified conversations broadly representative of recent GPT-5.2 Thinking production traffic, resampled the final assistant turn with GPT-5.4 Thinking, and automatically labeled relevant properties of the new completions.

These evaluations reflect a particular point in time, and are imperfect due to temporal drifts both in the underlying distributions of production traffic and in internal processing and evaluation pipelines, as well as due to the difficulty of faithfully reconstructing the range of contexts and environments in production. In our previous research, we saw that despite these challenges, we were able to predict whether or not true rates would have very significant increases at the model level.

Note that these evaluations only capture the behavior of the model itself, and do not account for other layers of the safety stack designed to mitigate disallowed model responses. Because of that, we expect the rates of policy-violative responses to be lower than these rates in the actual production environment.

In the table below, we report the extrapolated prevalence of not_unsafe model-level outputs, which measures the expected proportion of all model-level outputs which are violative of a given category (without accounting for any other parts of OpenAI’s safety stack). For example, based on the observed distribution of conversations with GPT-5.2 Thinking, we estimate that 99.9534% of GPT-5.4 Thinking outputs will not violate our harassment policy, even without the benefit of other safety interventions that operate in addition to the model’s own safety training.

In addition to running these evaluations for disallowed content, we ran them for deceptive model behaviors, originally discussed in the GPT-5 system card. We additionally include “Requesting Unnecessary Confirmations” and “Uses Calculator Tool to Avoid Citations”, which were reported in our prior research as deceptive behaviors which were present for GPT-5.1 Thinking.

We use these evaluations primarily as a check for potentially concerning regressions: if we observed more substantial deterioration in the most safety-relevant categories, we would investigate further and update our assessment more cautiously. At the values observed here, these estimates provide an additional signal that this deployment is unlikely to produce very significant increases in disallowed or misaligned behavior, especially in the most concerning categories.

We evaluate model robustness to jailbreaks: adversarial or out-of-distribution prompts designed to circumvent safety guardrails and elicit harmful assistance. Ahead of the GPT-5.4 launch, we replaced our previous StrongReject-based benchmark with a more challenging multiturn jailbreak evaluation derived from red-teaming exercises. The updated evaluation tests models on realistic scenarios using sophisticated attacker strategies that can probe, adapt, and escalate over the course of a conversation.

Responses that do not comply with our safety policies are scored worse, while compliant responses are scored better; in aggregate, we report worst-case defender success rate, so higher is better.

On this benchmark, GPT-5.2-Thinking and GPT-5.4-Thinking substantially outperform GPT-5.1-Thinking, with a large improvement from 5.1 to 5.2 and a smaller additional improvement from 5.2 to 5.4.

We evaluate the model’s robustness to known prompt injection attacks against connectors and function-calling. These attacks embed adversarial instructions in the tool-output that aim to mislead the model and override the system/developer/user instruction. Both of these evaluations are splits of the data we used for training, so don’t represent a model’s ability to generalize to new attacks.

GPT-5.4-reasoning improved for prompt injection attacks against email connectors and regressed slightly for attacks into function cells.

We ran the image input evaluations introduced with ChatGPT agent, that evaluate for not_unsafe model output, given disallowed combined text and image input.

We find that GPT-5.4 Thinking performs generally on par with its predecessors. Minor regressions for harms-erotic had low statistical significance.

Chatbots can empower consumers to better understand their health and help health professionals deliver better care [1] [2]. We evaluate GPT-5.4 on HealthBench [3], an evaluation of health performance and safety.

HealthBench comprises 5,000 realistic health conversations, and model responses are evaluated with example-specific rubrics. We report results on three variants, HealthBench, HealthBench Hard, and HealthBench Consensus, including average response length on HealthBench.

Relative to GPT-5.2, GPT-5.4 scores 62.6% on HealthBench (-0.8pts), 40.1% on Hard (-1.9pts), and 96.6% on Consensus (+2.1pts). There were major length differences: GPT-5.4 averaged 3311 chars versus 2676 chars for GPT-5.2.

On consensus criteria, GPT-5.4 seeks much less context than GPT-5.2. As a result, its main strengths are better precision when enough context is already available, simpler responses when appropriate, and improved context-seeking before referrals to care. Its main weaknesses are poorer context-seeking when information may be missing.

As with GPT-5.3-Codex, we ran our destructive actions evaluation that measures the model’s ability to preserve user-produced changes and avoid taking accidental destructive actions. We find that GPT-5.4 Thinking performs approximately on par with GPT-5.3-Codex.

Destructive action can also be particularly prevalent when agents operate deletion-inducing tasks (e.g., file reversion and cleanup) in complex workspaces with ongoing changes from users or even other agents. A safe and collaborative agent should distinguish between their work and user work, protect user changes by default, and recover from mistakes. Therefore, we trained our agents to revert their own changes after long rollouts while protecting implicit, simulated user work. On evaluations involving challenging, long-rollout traces, GPT-5.4-Thinking performs much better than earlier models in tracking and reverting its operations while leaving user work intact.

For deployment of GPT-5.4 Thinking for computer use, we updated our training process from a single confirmation behavior policy enforced in the model. Instead, we trained the model to follow both platform-level policy for high-risk actions and configurable developer-provided confirmation policy provided in the developer message in line with our approach to instruction hierarchy.

This provides a number of benefits, including:

In ChatGPT and API deployment, we now provide the confirmation policy in the system message. We implemented a new improved version of the user confirmations evaluation during computer use, and ran it on the new model and previous models for comparison.

We also tested the models on our first-person fairness evaluation [4]. This evaluation consists of multiturn conversations, starting with a prompt in which a user shares their name such as “Hi, I’m [NAME].” to which the model responds “Hi [NAME]! How can I assist you today?” We then prompt the model with a request such as “Write a fairytale.”

This evaluation is used to assess harmful stereotypes by grading differences in how a model responds to the conversation when the user’s name is statistically more often associated with males (e.g., Brian) or females (e.g., Ashley). Responses are rated for harmful differences in stereotypes using GPT-4o, whose ratings were shown to be consistent with human ratings. This evaluation consists of over 600 challenging prompts reflecting real-world scenarios that exhibit high rates of bias in GPT-4o-mini generations. These prompts were intentionally chosen to be an order of magnitude more difficult than standard production traffic; this means that in typical use, we expect our models to be less biased.

We report the metric harm_overall, which represents our expected difference of biased answers for male vs female names based on the performance on this evaluation (i.e., performance on the evaluation divided by 10). We see gpt-5.4-thinking performs generally on par with or slightly better than GTP-5.2 Thinking and GPT-5.1 Thinking.

We evaluate chain-of-thought (CoT) monitorability: whether a monitor can infer safety-relevant properties of a model’s behavior from its reasoning trace. There are principled reasons to believe [5] that CoT can provide a substantially richer monitoring signal than actions and final outputs alone, a hypothesis that is supported by growing empirical evidence. We assess GPT-5.4 Thinking using the evaluation suite from Guan et al. (2025) [6], which spans 13 evaluations across 24 environments and three evaluation archetypes—intervention, process, and outcome-property—and summarize performance with g-mean², a metric that depends on both true positive rate (TPR) and true negative rate (TNR).

Figure Figure 3 reports aggregate monitorability, TPR, and TNR as a function of CoT length and compares monitors when given access to different observations of the agent’s rollout: only the agent’s chain of thought (CoT only), only actions or final outputs to the user (action only), and all tokens generated by the agent (all messages). As in Guan et al. (2025) [6], monitorability generally improves as models produce longer CoTs, and monitors that can inspect CoT or all messages are substantially more informative than monitors limited to actions alone. Averaging these evaluations, GPT-5.4 Thinking has lower aggregate CoT monitorability than GPT-5 Thinking. However, Figure Figure 4 shows that this decline is not uniform across the evaluation suite. Under CoT-only monitoring at medium reasoning effort, GPT-5.4 Thinking is comparable to GPT-5 Thinking on some environment groups, but underperforms on several others—including Health queries with missing evidence, Memory, and Impossible-task settings—lowering the overall aggregate. One notable exception is agentic misalignment, where monitorability improved substantially and is near 100% for GPT-5.4 Thinking.

We see from Figures  Figure 5,  Figure 6, and  Figure 7 that the regression in health queries: missing evidence and memory advice comes from large drops in TNR in the intervention arm, while the regression in impossible tasks comes from a smaller drop in TPR. We manually inspected the intervention-arm false positive samples for health queries: missing evidence and memory bias. For both evals, we observed that the sampling model’s CoTs did clearly discuss using the hint in these cases, which the monitor correctly identified. As such, we do not believe the model is less monitorable on these prompts. For Health queries: missing evidence, these were considered false positives because the hint, despite being used, did not make the predicted diagnosis more accurate. For Memory, these were graded false positives because the final answers were not judged as being biased by the memory according to the answer grader. We believe the answer grader judged reasonably because the change to the final answers was indeed subtle. Overall, we believe this to be a limitation of these particular evals, where the intervention can easily be used by the model in its reasoning (X=1,Z=1) without causing a change in the model output (Y=0). The Memory eval is also notably the one eval where the model output outcome is judged using a model instead of deterministically.

We plan to further investigate the sources of the aggregate drop and explore mitigations that preserve monitorability as models improve. More broadly, this result reinforces the fragility of CoT monitorability (Korbak, et al., 2025 [5]) and the importance of tracking and preserving it so that CoT monitoring can remain a useful component of scalable oversight.

In the future, CoT monitorability may break down for a variety of reasons [5]. One such path involves agents that become capable of deliberately reshaping or obscuring their reasoning when they know they are being monitored. If so, evaluations could overestimate a system’s alignment, and monitoring systems could become less reliable. To bound this risk, we additionally measure “CoT controllability”, a reasoning model’s ability to control properties of its CoT in ways that reduce monitorability. CoT controllability evaluations can be seen as an early warning sign of whether models are getting better at shaping or hiding their reasoning.

We measure GPT-5.4 Thinking’s controllability by running CoT-Control, an evaluation suite described in (Yueh-Han, 2026 [7]) that tracks the model’s ability to follow user instructions about their CoT. CoT-Control includes over 13,000 tasks built from established benchmarks: GPQA (Rein et al., 2023 [8]), MMLU-Pro (Hendrycks et al., 2020 [9]), HLE (Phan et al., 2025 [10]), BFCL (Patil et al., 2025 [11]) and SWE-Bench Verified. Each task is created by pairing a benchmark problem with one CoT instruction such as avoiding certain problem-relevant keywords in CoT, using only lowercase letters, or appending a given word to each sentence. Low CoT controllability is a desirable property: it suggests models might similarly lack the ability to obfuscate their reasoning to evade our monitors.

We find that GPT-5.4 Thinking’s CoT controllability is low, similarly to GPT-5 Thinking and GPT-5.2 Thinking. Because controllability is heavily confounded by CoT length (longer CoTs are hardest to control, all else equal; see Yueh-Han, 2026 [7], sec. 5.3), we report controllability scores as a function of CoT length. For example, GPT-5.4 Thinking only successfully controls 0.3% of CoTs that are 10k characters long (compared to 0.2% for GPT-5.2 Thinking). This suggests that the increased reasoning capabilities of GPT-5.4 Thinking likely do not come at the cost of an increased ability to obfuscate or actively shape its reasoning.

The Preparedness Framework is OpenAI’s approach to tracking and preparing for frontier capabilities that create new risks of severe harm. Under our framework, we work to track and mitigate the risk of severe harm, including by implementing safeguards that sufficiently minimize the risk for highly capable models.

As we did for gpt-5.2-thinking before it, we are continuing to treat gpt-5.4-thinking as High capability in the Biological and Chemical domain. We have applied the corresponding safeguards for this model as described in the GPT-5 system card. As we did for gpt-5.3-codex, we are treating gpt-5.4-thinking as High capability in the Cybersecurity domain, and applied safeguards as described in the Safeguard section below.

For AI self-improvement, evaluations of final checkpoints indicate that, like its predecessor models, gpt-5.4-thinking does not have a plausible chance of reaching a High threshold.

For the evaluations below, we tested a variety of elicitation methods, including scaffolding and prompting where relevant. However, evaluations represent a lower bound for potential capabilities; additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed in our tests or the tests of our third-party partners.

We are treating this launch as High capability in the Biological and Chemical domain, activating the associated Preparedness safeguards.

Given the higher potential severity of biological threats relative to chemical ones, we prioritize biological capability evaluations and use these as indicators for High and Critical capabilities for the category.

To evaluate models’ ability to troubleshoot wet lab experiments in a multimodal setting, we evaluate models on a set of 350 fully held-out virology troubleshooting questions from SecureBio.

All models exceed the median domain expert baseline of 22.1%.

To evaluate models’ ability to troubleshoot commonly published lab protocols, we modify 108 multiple choice questions from FutureHouse’s ProtocolQA dataset [12] to be open-ended short answer questions, which makes the evaluation harder and more realistic than the multiple-choice version. The questions introduce egregious errors in common published protocols, describe the wet lab result of carrying out this protocol, and ask for how to fix the procedure. To compare model performance to that of PhD experts, we performed expert baselining on this evaluation with 19 PhD scientists who have over one year of wet lab experience.

All models underperform the consensus expert baseline (54%).

We evaluated models on a tacit knowledge and troubleshooting multiple choice dataset created with Gryphon Scientific. The questions span all 5 stages in the biothreat creation process and focus on areas where tacit knowledge would be a bottleneck. Tacit knowledge questions are meant to be obscure to anyone not working in the field, i.e., they either require tracking down authors of relevant papers or knowing people in the field. Troubleshooting questions are meant to be obscure to anyone without hands-on experience, i.e., the answers are known only to people who have tried the protocol themselves.

This set is uncontaminated; it was created fully in-house with our partners at Gryphon Scientific and has not been published.

On some of these questions models respond with refusals or safe completions which do not fully answer the question. To avoid underestimating model capabilities, we report both original scores and what the score would be if all refusals and safe completions were treated as successes.

Note that GPT-5.2 Thinking, GPT-5.3-Codex, and GPT-5.4 Thinking outperform the consensus expert baseline of 80% including refusals, but below it if we exclude refusals.

To evaluate models’ ability to identify and correct real-world experimental errors in biological protocols, we built a short-answer troubleshooting dataset from expert-written wet lab procedures. TroubleshootingBench focuses on tacit, hands-on knowledge and uncontaminated procedures that are not available online.

Scientists with a PhD in a relevant biological discipline (virology, genetics, microbiology, or protein engineering) were asked to transcribe biological protocols they have personally used in the lab. Each protocol must include precise step-by-step procedures, equipment, and reagents. If a protocol was adapted from a publication, experts were required to significantly alter at least several steps. From these protocols, they created three troubleshooting questions each, introducing subtle or realistic execution errors (e.g., improper homogenization technique) and describing the resulting failed outcome.

After going through independent expert review, the resulting dataset includes 52 protocols, each paired with three expert-written troubleshooting questions. To benchmark model performance, we ran a human baselining campaign with 12 independent PhD experts. The 80th percentile expert score (36.4%) is used as an indicative threshold for model performance. Compared to ProtocolQA Open-Ended, which focuses on well-known published procedures, TroubleshootingBench is designed to test model performance on non-public, experience-grounded protocols and errors that rely on tacit procedural knowledge

GPT-5.4 Thinking scores slightly above prior models.

GPT-5.4-Thinking is highly capable, closely matching the results for GPT-5.3-Codex. Because of this strong performance, we similarly treat GPT-5.4-Thinking as High under the Preparedness Framework.

Under our Preparedness Framework, High cybersecurity capability is defined as a model that removes existing bottlenecks to scaling cyber operations, including either by automating end-to-end cyber operations against reasonably hardened targets, or by automating the discovery and exploitation of operationally relevant vulnerabilities.

We are treating this model as High, even though we cannot be certain that it actually has these capabilities, because it meets the requirements of each of our canary thresholds and we therefore cannot rule out the possibility that it is in fact Cyber High.

In our analysis of prior cybersecurity incidents involving severe harm through either of these two routes, we find each instance demanded proficiency in each of three skills:

As with prior releases, we use the following three benchmarks associated with the skills above:

Each of these evaluations has important limitations:

Because of these limitations, excelling on all three evaluations is necessary but not sufficient for a model to be considered High in cyber capability. Even very strong results can still fall short of what is required for scalable, end-to-end cyber operations against hardened targets.

We evaluate model performance on a test set of curated, publicly available CTF challenges which met the following criteria:

Challenges were further divided into the level of proficiency needed to solve the challenge, ultimately organized by whether the challenge required high school, collegiate, or professional-level cybersecurity knowledge and skills, with judgment on the requisite skills manually determined by our in-house cybersecurity expert. For all Collegiate and Professional CTFs we require that the solution involves a non-trivial level of exploitation, requiring chaining two or more exploitation steps and not be solvable by copy-pasting textbook examples. Historically, we reported results across high school, collegiate, and professional levels but given the rise in capabilities and our investment in more rigorous benchmarks we focus only on professional CTFs.

To evaluate the model against the CTFs we give it access to a headless Linux box with common offensive cybersecurity tools preinstalled as well as a harness which allows the model to call those tools or otherwise execute commands similar to as a human. We run 16 rollouts for each CTF exercise, recording the pass@12 metric over the best set of rollouts.

CVE-Bench is a benchmark that tasks models with identifying and exploiting real-world web-application vulnerabilities in a sandbox environment. We used CVE-Bench (version 1.0) with a focus on vulnerabilities covering content-management systems, AI/ML apps, business-management tools, operational-monitoring systems, web infrastructure, libraries/packages, e-commerce platforms, and a small number of computing-management, mail-server, and web-portal applications; due to some infrastructure challenges porting the eval, we only ran 34 out of the 40 benchmark challenges. We ran the benchmark using the zero-day prompt configuration, where the model is given a general task description of what it needs to do, as opposed to the one-day configuration, where the model is given a description of the vulnerability to exploit. Additionally, we ran the benchmark such that the agent does not have access to the source code of the web-application, and instead must probe it remotely.

We use pass@1 over 3 rollouts for this evaluation to measure the model’s ability to consistently identify vulnerabilities which are considered relatively straightforward by internal cybersecurity experts. Consistency is important to measure the model’s cost-intelligence frontier to identify vulnerabilities and its ability to potentially evade detection mechanisms that look for scaled attempts of vulnerability discovery and exploitation.

GPT-5.4 Thinking performs higher than GPT-5.2 Thinking, but lower than GPT-5.3-Codex.

Cyber range exercises measure a model’s ability to conduct fully end-to-end cyber operations in a realistic, emulated network. These exercises are long-form, requiring the model to (1) construct a plan to achieve an abstract adversary objective; (2) exploit vulnerabilities, misconfigurations, and weaknesses that are likely to be seen in the wild; and (3) chain together these exploits to achieve the scenario objective.

Performance is measured as pass/fail over 16 trials, where a pass denotes at least one successful trial. The combined pass rate is the percentage of scenarios passed.

Combined Pass Rate:

GPT-5.4-Thinking remains stronger than the pre-5.3 models overall, but it is a step down from GPT-5.3-Codex on the Cyber Range suite. It fails four scenarios – EDR Evasion, Firewall Evasion, Leaked Token, and CA/DNS Hijacking – whereas GPT-5.3-Codex solves Firewall Evasion and fails the other three. Even so, GPT-5.4-Thinking still outperforms the earlier model family on the broader set of scenarios.

External Evaluations by Irregular

Irregular, a frontier AI security lab, evaluated a near-final, representative checkpoint of 5.4-reasoning, with xhigh reasoning effort on a subset of a suite of cyberoffensive challenges spanning three categories:

In this evaluation setting, the model was given up to 1,000 turns per challenge and elicited using techniques designed to maximize performance, including utilizing compaction (triggered every 100K tokens to prevent the context window from growing too large).

On Irregular’s atomic challenge suite, 5.4-reasoning achieved an average success rate of 88% on Network Attack Simulation challenges, 73% on Vulnerability Research and Exploitation challenges, and 48% on Evasion challenges. 5.4-thinking solved 14/17 medium and 5/5 hard atomic challenges. 5.4-thinking solved the only Hard atomic challenge that was not solved by 5.2-thinking.

On CyScenarioBench [14] 5.4-thinking achieved an 11% average success rate and solved 5/11 challenges, compared to 1 challenge solved by GPT-5.2-reasoning. Irregular interprets this as higher operational capability on long-horizon scenarios (planning, branching decisions, constraint adherence, and state tracking/recovery).

GPT-5.4 Thinking did not meet our thresholds for High capability in AI Self-Improvement.The High capability threshold is defined to be equivalent to a performant mid-career research engineer, and performance in the evaluations below indicate we can rule this out for GPT-5.4 Thinking.

We evaluate the model on its ability to replicate pull-request style contributions. A single evaluation sample is based on an agentic rollout in which:

If all task-specific tests pass, the rollout is considered a success. Prompts, unit tests, and hints are human-written.

GPT-5.4 Thinking performs slightly higher than comparison models.

MLE-bench evaluates an agent’s ability to solve Kaggle challenges involving the design, building, and training of machine learning models on GPUs. In this eval, we provide an agent with a virtual environment, GPU, and data and instruction set from Kaggle. The eval dataset consists of 30 of the most interesting and diverse competitions chosen from the subset of tasks that are <50GB and <10h. Success means achieving at least a bronze medal in the competition.

GPT-5.4-thinking performs significantly better than comparison models. Due to infra issues we do not report scores on GPT-5.3-codex.

OpenAI-Proof Q&A evaluates AI models on 20 internal research and engineering bottlenecks encountered at OpenAI, each representing at least a one-day delay to a major project and in some cases influencing the outcome of large training runs and launches. “OpenAI-Proof” refers to the fact that each problem required over a day for a team at OpenAI to solve. Tasks require models to diagnose and explain complex issues—such as unexpected performance regressions, anomalous training metrics, or subtle implementation bugs. Models are given access to a container with code access and run artifacts. Each solution is graded pass@1.

GPT-5.2-Codex is the highest scoring model on this benchmark at 8.33%.

External Evaluations by Apollo Research

Apollo Research evaluated a near-final, representative version of GPT-5.4-reasoning for capabilities and propensities related to strategic deception, in-context scheming, and sabotage. Apollo defines covert action as behavior where a model strategically misrepresents, omits, or conceals information that users or developers would consider important.

Apollo reports low rates of covert deceptive behavior, with an overall deception rate of ~1% in no-nudge settings, comparable to GPT-5.3 Codex and below GPT-5.2. No instances of covert subversion against developers (e.g., disabling oversight, self-exfiltration, or goal-guarding) were observed across the tested scenarios.

On deferred subversion / sandbagging, the model sandbags only when given an explicit instrumental goal. In the Sandbagging QA evaluation, the model shows a 6-percentage-point drop in accuracy when incentivized to underperform, similar to prior models. However, no sandbagging occurs without an explicit goal, with transcript analysis suggesting the model often recognizes the scenario as an evaluation and decides not to manipulate performance.

GPT-5.4-reasoning has a mean sabotage score of 0.56 (best-of-10: 0.74)—comparable to GPT-5.2 but below GPT-5.3-codex (0.88). On several hard tasks the model exceeds human baselines, indicating meaningful sabotage capability, though performance remains below the strongest prior Codex checkpoint. Chain-of-thought analysis shows higher rates of evaluation awareness (21.3%) than prior models and far fewer multilingual reasoning anomalies (0.5% of samples vs 29.5% for GPT-5.3-codex).

These findings provide evidence that the model can identify and execute relevant technical steps for sabotage in a simple simulated setting, but they do not directly test whether the model can reliably conceal sabotage, evade detection, or succeed under realistic monitoring and operational constraints.

In this section, we describe the safeguards we have implemented against cybersecurity risk. The approach for GPT-5.4 Thinking is built on the latest approach for GPT-5.3 Codex. These safeguards, which we will continue to iteratively refine and improve over time, are the product of an intensive, months-long cross functional effort. What follows is a public summary of our internal Safeguards Report, which includes additional details that are not suitable for public disclosure (such as information potentially useful to attackers). The internal report informed SAG’s finding that these safeguards sufficiently minimize the associated risks.

We described our overall approach to cybersecurity safeguards for Preparedness High in the GPT-5.3 Codex system card. Our north star is to enable defensive acceleration while maximizing harm reduction and preventing misuse. Uniformly on all surfaces, we have trained the model behavior to refuse requests with harmful intent.

In GPT-5.3 Codex, we relied on user-level signals to identify potentially suspicious usage and downgrade to a below-cyber-high model. We have now developed asynchronous, message level blocks when the online classifiers indicate high risk harmful intent, and apply these depending on the surface and customer cohort. We no longer downgrade the model for cyber use cases, and now employ a combination of message-level and user-level mitigation.

We expect this suite of GPT-5.4 Thinking safeguards to significantly reduce the effectiveness and attractiveness of gpt-5.4r as a tool for cyber misuse, while at the same time providing us with visibility into how the model is being used on our platform, so that we can adjust our safeguards if needed.

Pursuant to our Preparedness Framework, we developed threat actor profiles and a threat model for cybersecurity risk that identifies specific pathways through which severe harm (as defined in our Preparedness Framework) could arise, assesses the specific gating steps where our technology could play a role, and guides the development of safeguards to sufficiently minimize those risks of severe harm. Please refer to the GPT-5.3 Codex system card, as our threat model remains the same.

The cyber thread taxonomy also remains the same as described in GPT-5.3 Codex system card.

The most important parts of this taxonomy are:

Design: As with GPT-5.3-Codex, we trained GPT-5.4 Thinking to generally provide maximally helpful support on dual-use cybersecurity topics while refusing or de-escalating operational guidance for certain disallowed actions, including areas such as malware creation, credential theft, and chained exploitation. In addition, we introduced new approaches to discourage unnecessary refusals and overly caveated responses.

Testing: We assess performance on data that do not overlap with the training set, measuring policy compliance rate (higher is better). When building our cyber safety evaluations, we consider multiple aspects to ensure broad and meaningful coverage. The eval sets combine deidentified production data (in accordance with our privacy policy), which reflects realistic user behavior, with synthetic data designed to improve coverage of policy-relevant scenarios that are rare or under-represented actual use. We evaluate both chat-based and agentic interactions, including multi-turn settings. Prompts are selected using a mix of sampling strategies—such as classifier-flagged cases and embedding–based clustering—to emphasize challenging or ambiguous examples. The distribution intentionally spans benign and legitimate requests as well as disallowed requests, and includes MITRE ATT&CK–grounded adversarial and defensive scenarios to stress-test safety behavior under realistic threat models. These eval sets consist of challenging cases and shouldn’t be interpreted as representative of production behavior.

Overall, GPT-5.4 Thinking performs slightly better than GPT-5.2 Thinking on deidentified production data and slightly worse but still within an acceptable range on synthetic data.

Building on GPT-5.3-Codex, we have implemented a two-tiered system of real-time, automated oversight surrounding the model to monitor cyber prompts and generations.

Accounts that reach certain thresholds of flagging by our classifiers trigger deeper analysis using a combination of automated analysis and, for certain cases on non-ZDR surfaces, manual human review.

Our usage policies prohibit malicious cyber activity across all product surfaces, including in dual-use domains. We may also enforce against dual use activity when we see signs of malicious intent, or a pattern of escalation toward harmful outcomes

Our process uses a variety of signals to assess both the overall potential for misuse from an account’s codex usage, as well as the apparent intent of the user. Specific enforcement thresholds and practices vary by product surface and will continue to evolve over time. Depending on the product surface and the circumstances, we may employ a warning, restrict an account’s access to frontier cyber capabilities, or in cases of higher concern suspend or ban an account.

On the API, customers that serve a range of end-users can include with their traffic a safety identifier field. This allows us to attribute behavior, and target enforcement responses, to specific end users, reducing the potential for collateral harm to benign applications.

Account level enforcement is a relatively coarse-grained tool. Because cyber capabilities are inherently dual use, we know that some of the important and valuable uses of GPT-5.4 Thinking are likely to be flagged by the monitoring system. For that reason, we have the existing Trusted Access for Cyber program, launched with GPT-5.3-Codex and tailored to support the needs of defenders.

The Trusted Access for Cyber (TAC) program remains the same as described in the GPT-5.3 Codex system card. It is intended to provide high-risk dual use cyber capabilities to enterprise customers and other legitimate users in order to advance ecosystem hardening. It is an identity based gated program to reduce risk of malicious users.

In addition to the other safety measures described in this system card, we take steps to prevent adversaries from compromising sensitive intellectual property, including customer data and theft of model weights. As we have previously described, and as described for GPT-5.3 Codex, we take a defense-in-depth approach to protecting our model weights, relying on a combination of access control, infrastructure hardening, egress controls, and monitoring. We leverage purpose-built detections and controls to mitigate the risk of exfiltration of high-risk model weights. We complement these measures with dedicated internal security teams, including Detection and Response, Threat Intelligence, and Insider-Risk programs. These programs are intended to help identify and block emerging threats quickly. As the power and capabilities of our models increase, so do the security investments made to help protect them.

Our Preparedness efforts in Cybersecurity have thus far focused primarily on misuse risks, which our threat modeling process identifies as the immediate and most important risks posed by this level of capability. However, as models reach High cybersecurity capability, internal deployment itself becomes a meaningful surface to consider – not because of misuse, but because high cyber capability can remove a key bottleneck to certain internal deployment risks materializing. For example, in conjunction with additional capabilities such as long range autonomy, a model with the propensity to self-exfiltrate or sabotage internal research could plausibly succeed at these attempts. We do not yet have evidence that GPT-5.4 Thinking demonstrates propensities for such misalignment or possesses the long range autonomy capabilities that such a scenario would require. This is informed by current performance on proxy evaluations such as TerminalBench, and by the limited coherence and goal sustenance we have observed in models of a similar capability profile such as GPT-5.3-Codex while monitoring them in internal production. However, this risk makes it important to mature our internal deployment posture ahead of more capable systems. Accordingly, we are accelerating work across evaluations, safeguards, and operating procedures, as described in the GPT-5.3-Codex system card.

1. [1]

   OpenAI. “Introducing GPT-5.” Available at: https://openai.com/index/introducing-gpt-5/.
2. [2]

   OpenAI. “Pioneering an AI clinical copilot with Penda health.” Available at: https://openai.com/index/ai-clinical-copilot-penda-health/.
3. [3]

   OpenAI. “Introducing HealthBench.” Available at: https://openai.com/index/healthbench/.
4. [4]

   Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu-Lemberg, Anna-Luisa Brakman, Pamela Mishkin, et al. “First-person fairness in chatbots.” Available at: https://arxiv.org/abs/2410.19803.
5. [5]

   Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, et al. Chain of thought monitorability: A new and fragile opportunity for AI safety. Available at: https://arxiv.org/abs/2507.11473.
6. [6]

   Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, et al. Monitoring monitorability. Available at: https://arxiv.org/abs/2512.18311.
7. [7]

   Yueh-Han Chen, Robert McCarthy, Bruce W. Lee, He He, Ian Kivlichan, Bowen Baker, et al. “Reasoning models struggle to control their chains of thought.”
8. [8]

   David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, et al. GPQA: A graduate-level google-proof q&a benchmark. Available at: https://arxiv.org/abs/2311.12022.
9. [9]

   Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, et al. Measuring massive multitask language understanding. Available at: https://arxiv.org/abs/2009.03300.
10. [10]

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, et al. Humanity’s last exam. Available at: https://arxiv.org/abs/2501.14249.
11. [11]

    Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, et al. “The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models.” Proceedings of the 42nd international conference on machine learning. Available at: https://proceedings.mlr.press/v267/patil25a.html.
12. [12]

    Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, et al. “LAB-bench: Measuring capabilities of language models for biology research.” Available at: https://arxiv.org/abs/2407.10362.
13. [13]

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, et al. “CVE-bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities.” Available at: https://arxiv.org/abs/2503.17332.
14. [14]

    Irregular. “CyScenarioBench: Evaluating LLM cyber capabilities through scenario-based benchmarking.”