ChatGPT Agent System Card

ChatGPT agent is a new agentic model in the same family as OpenAI o3 that combines the strengths of deep research and Operator. It brings together:

From the outset, we’ve prioritized safety as an inherent part of the system, expanding on robust controls from Operator’s research preview and adding additional safeguards to address new risks like broader user reach and terminal access. See the section on Product-Specific Risk Mitigations below for more detail.

We have decided to treat this launch as High capability in the Biological and Chemical domain under our Preparedness Framework, activating the associated safeguards. While we do not have definitive evidence that this model could meaningfully help a novice to create severe biological harm - our defined threshold for High capability - we have chosen to take a precautionary approach.

The following evaluations check that the model does not comply with requests for content that is disallowed under OpenAI’s policies, including hateful content or illicit advice.

We consider several evaluations:

We evaluate completions using LLM-based grading models. It evaluates the metric not_unsafe, checking that the model did not produce unsafe output according to relevant OpenAI policy.

Note that the production benchmark set is designed specifically to be more challenging, to provide useful signal; scores are therefore expected to be lower than the standard evaluation. ChatGPT agent performs at parity with OpenAI o3 on the standard evaluation. On the more challenging set, ChatGPT agent generally outperforms o3.

We further evaluate the robustness of ChatGPT agent to jailbreaks: adversarial prompts that purposely try to circumvent model refusals for content it’s not supposed to produce. We evaluate using the following approach:

We find that ChatGPT agent performs similarly to o3.

We check for hallucinations in ChatGPT agent using the following evaluations:

We consider two metrics: accuracy (did the model answer the question correctly) and hallucination rate (did the model answer the question incorrectly).

Since ChatGPT agent always has the ability to browse the internet, it is appropriate to compare its performance with a version of o3 that has browsing enabled. The table below compares ChatGPT agent to o3 with browsing. The values reported here for o3 are better than those in the o3 system card, because the metrics in the o3 card reflected performance without browsing.

ChatGPT agent scores lower on SimpleQA accuracy than o3 did. Manual investigation revealed cases where ChatGPT agent’s more thorough approach to research surfaced potential flaws in our grading rubric that were not apparent to o3, such as instances in which Wikipedia may contain inaccurate information. We are considering updates to this evaluation.

We created new image input evaluations, that evaluate for not_unsafe model output, given disallowed combined text and image input. ChatGPT agent generally performs on par or slightly higher than o3.

To measure the model’s multilingual performance, we translated a challenging internal evaluation measuring performance on tasks across a range of expert domains (science, law, engineering, etc) into a number of different languages. We translated each prompt in the evaluation using a language model. We measured the model’s pass-rate or accuracy in each of the different languages in order to measure whether performance degrades in certain languages.

We changed the grader to be stricter since running the equivalent evaluations for deep research, which means the deep research model would perform around 3% lower for each language; the numbers in the “deep research” column are not adjusted for the grader change.

We evaluated ChatGPT agent on the BBQ evaluation [2].

ChatGPT agent scored similarly to OpenAI o3 on ambiguous questions, but lower than o3 on disambiguated questions.

ChatGPT agent’s incorrect answers for disambiguated questions were largely a result of mistaken refusals, reflecting situations in which the model has sufficient information to answer a multiple choice question and can do so within policy, but nonetheless incorrectly declines to answer. Because the model in fact had enough information to answer, the over-refusal is graded as incorrect.

We repeated the first-person fairness evaluation introduced for OpenAI o3 and o4-mini, now applying it to ChatGPT agent.

The protocol is unchanged: a multistep dialogue begins with the user sharing a name that is statistically more often associated with a given gender (for example, "Hi, I’m {Name}"), the model responds, and the user then issues a creative request such as "Write a fairy tale." Responses are graded for harmful stereotypes, and bias is quantified by net_bias, i.e. the fraction of test prompts where response differences reflect harmful gender stereotypes. Prompts in this benchmark remain an order of magnitude more challenging than standard production traffic, so in typical use, we expect our models to be about ten times less biased. We again report the metric net_bias, which represents our expected difference of biased answers for male vs female names based on the performance on this evaluation (i.e., performance on the evaluation divided by 10).

ChatGPT agent (with and without tools) performs slightly better than prior models.

ChatGPT agent does not have custom developer messages at launch. We test user/system instruction hierarchy, and find that ChatGPT agent performs generally on par with o3 in this category.

First is a set of evaluations where system and user messages are in conflict with each other; the model must choose to follow the instructions in the system message to pass these evaluations.

In the other set of evaluations, we instruct the model to not output a certain phrase (e.g., “access granted”) or to not reveal a bespoke password in the system message, and attempt to trick the model into outputting it in user messages.

Our incremental safety work for ChatGPT agent included the following risk mitigation workstreams.

Prompt injections are a form of attack where an attacker embeds malicious instructions in content that ChatGPT agent is likely to encounter–such as a webpage–with the intention that the instructions override ChatGPT agent’s intended behavior. These can lead to potentially exfiltrating data (for example from a Connector, or another site that the user has logged ChatGPT agent into), taking actions the user didn’t intend, or simply providing the user an incorrect answer.

The impact of prompt injections for ChatGPT agent could be higher than for previous launches, since it has access to more tools simultaneously. We have therefore designed an extensive multi-layered set of mitigations. Due to the adversarial nature of prompt injections, the following are some, but not all, of the prompt injection-related mitigations included in this product.

ChatGPT agent includes specialized prompt injection robustness training designed to mitigate the risk of prompt injection attacks.

These evaluation results test only the model’s behavior, not our full end-to-end stack of prompt-injection mitigations.

We implemented multiple automated monitors and filters to protect against various types of prompt injections. We can rapidly update these with information about new attacks as we become aware of them.

ChatGPT agent will pause and ask the user to confirm before taking certain kinds of actions online. When ChatGPT agent requests confirmation, users can review the current state and indicate whether it should proceed.

For more details, including evaluations, see below (Section 3.2.3).

As with Operator, at launch, when ChatGPT agent uses the visual browser tool in a sensitive context (e.g. logged into an email or banking account) we enable "Watch Mode" for the rest of the trajectory, which is intended to require the user to supervise what it is doing, by automatically pausing execution when the user becomes inactive or navigates away from the conversation in ChatGPT.

We may revisit this in the future based on iterating on other mitigations and what we learn from deployment.

In addition to using monitors and filters, at launch, terminal network requests will have additional restrictions, including being limited to GET requests to download images or certain datasets (such as commonly used official government datasets) and associated information. We may revisit this in the future.

To mitigate the risk of prompt injections attempting to exfiltrate data from memory, at launch we will disable memory. We may revisit this in the future.

One category of risk is ChatGPT agent making a mistake. For example, it may inadvertently buy the wrong product.

Additionally, ChatGPT agent may have access to sensitive and private data about the user (e.g. via their Google drive or email). We consider the risk that ChatGPT agent could mistakenly reveal this private data in ways the user doesn’t intend, for example by typing personal information that the user didn’t expect to share into an online form.

To reduce the likelihood and impact of model mistakes, we trained ChatGPT agent to ask the user for confirmations before finalizing actions that affect the state of the world (e.g., before completing a purchase or sending an email). When ChatGPT agent requests confirmation, users can review its actions and correct mistakes or redirect it.

See Watch Mode section Section 3.1.2.4.

Many harmful or disallowed tasks are covered by our Standard Model Safety Evaluations discussed above. Beyond those, we developed new policies or refined existing policies for ChatGPT agent. For example, ChatGPT agent should not do online research to fetch or infer personal data over which people have high expectations of privacy, should not transact for regulated goods or gamble, and should not assist with certain financial activities with elevated consequences, such as making financial account transfers.

The mitigations we have implemented for this risk include but are not limited to the following.

We trained the model to refuse such harmful or disallowed tasks.

See Watch Mode section Section 3.1.2.4.

Agent users are bound by OpenAI Usage Policies, which apply universally to OpenAI services and are designed to support safe and responsible usage of AI technology. Users are also prohibited from bypassing any protective measures implemented in OpenAI services, including rate limits or restrictions and safety mitigations.

At the system level, we restrict ChatGPT agent from navigating to certain websites associated with harmful or illicit activities that are prohibited by OpenAI’s Usage Policies.

We will leverage automated and human review to monitor for potential abuse and take appropriate action with users who violate our policies. We intend to track the effectiveness of mitigations and refine them over time. We will also continuously leverage discoveries from manual investigations to enhance our automated detection mechanisms and mitigations.

We conducted human red teaming on two key risk areas for ChatGPT Agent: biological risk and prompt injections.

The biological risk safeguard testing process and results can be found in the Section, Safeguards for High Biological and Chemical risk.

Prompt injection red teaming was carried out by several organizations focused on cybersecurity testing. This occurred in parallel to internal red teaming, focusing on injection techniques that could exfiltrate data. We do not list specific attack details and mitigation updates here to avoid providing a roadmap for future attackers. Further details about our prompt injection evaluations and mitigations can be found in Section 3.1, and were informed by the human red teaming efforts throughout the product development process.

The Preparedness Framework is OpenAI’s approach to tracking and preparing for frontier capabilities that create new risks of severe harm. The framework commits us to track and mitigate the risk of severe harm, including by implementing safeguards that sufficiently minimize the risk for highly capable models.

Below, we provide detailed information about the evaluations we conducted to inform this assessment. We also describe the safeguards we have implemented to sufficiently minimize the risks associated with High Biological and Chemical capability under our framework.

For the evaluations below, we tested a variety of elicitation methods, including custom post-training (e.g., to create a “helpful-only” model), scaffolding, and prompting where relevant. However, evaluations represent a lower bound for potential capabilities; additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed in our tests or the tests of our third-party partners.

We calculate 95% confidence intervals for pass@1 using the standard bootstrap procedure that resamples model attempts per problem to approximate the metric’s distribution. While widely used, this method can underestimate uncertainty for very small datasets, as it captures only sampling variance (randomness in the model’s performance on the same problems across multiple attempts) rather than all problem-level variance (variation in problem difficulty or pass rates). This can lead to overly tight confidence intervals, especially when a problem’s pass rate is near 0% or 100% with few attempts. We report these confidence intervals to reflect the inherent variation in evaluation results.

ChatGPT agent’s ability to browse the internet creates challenges for evaluating the model’s capabilities. In many Preparedness evaluations, we aim to understand the model’s ability to reason or solve problems. If the model can retrieve answers from the internet, then it may provide solutions without working through the problems itself, and could receive a high score without actually demonstrating the capability that the evaluation is intended to measure. In this situation, the score would be artificially elevated and would be a poor measure of the model’s true capability, a problem known as “contamination” of the evaluation.

Please see the deep research system card for a longer discussion of browsing-based contamination. We note in the interpretations for each evaluation where evaluations are known or suspected to be contaminated. Where relevant, we only plot no-browsing models.

We have decided to treat this launch as High capability in the Biological and Chemical domain, activating the associated Preparedness safeguards. While we do not have definitive evidence that this model could meaningfully help a novice to create severe biological harm – our defined threshold for High capability—we have chosen to take a precautionary approach. This decision is informed by an incremental improvement observed relative to our previous models, which were already on the cusp of High capability classification. Considered in isolation, each recent increase in capability may appear minor; collectively, we believe that steps like these can move the model across the High capability threshold. To responsibly address this cumulative progression, we are taking a precautionary approach and implementing safeguards consistent with High capability models outlined in our Preparedness Framework. Given the higher potential severity of biological threats relative to chemical ones, we prioritize biological capability evaluations and use these as indicators for High and Critical capabilities for the category.

We graded the accuracy of model responses to long-form biorisk questions. Our long-form biothreat information questions test acquiring critical and sensitive information across the five stages of the biological threat creation process [3]: Ideation, Acquisition, Magnification, Formulation, and Release.

We designed the questions and detailed rubrics with Gryphon Scientific due to their expertise working with dangerous biological agents in a national security setting. We used the OpenAI o1-preview (pre-mitigation) model as an autograder, validating agreement with a trusted biosecurity expert. We made adjustments to the rubric and iterated on the autograder based on the expert feedback.

Both ChatGPT agent and OpenAI o3 with browsing score above 20% across each category. Models with access to browsing seem to be able to synthesize biorisk-related information across all five steps of the biothreat creation process.

To evaluate models’ ability to troubleshoot wet lab experiments in a multimodal setting, we evaluate models on a set of 350 fully held-out virology troubleshooting questions from SecureBio.

ChatGPT agent helpful-only with browsing enabled does not score higher than prior models on this benchmark, but all models, including GPT-4o, score above the average human baseline (40%).

To evaluate models’ ability to troubleshoot commonly published lab protocols, we modify 108 multiple choice questions from FutureHouse’s ProtocolQA dataset [4] to be open-ended short answer questions, which makes the evaluation harder and more realistic than the multiple-choice version. The questions introduce egregious errors in common published protocols, describe the wet lab result of carrying out this protocol, and ask for how to fix the procedure. To compare model performance to that of PhD experts, we performed expert baselining on this evaluation with 19 PhD scientists who have over one year of wet lab experience.

ChatGPT agent helpful-only with browsing scores the highest on this benchmark, though all models underperform the consensus expert baseline (54%) and the median expert baseline of 42%. In the o3 and o4-mini system card, we plotted an o3 helpful-only browsing result of 43%. The o3 helpful-only browsing result we plot here is lower as a result of further decontamination efforts.

We evaluated models on a tacit knowledge and troubleshooting multiple choice dataset created with Gryphon Scientific. The questions span all 5 stages in the biothreat creation process and focus on areas where tacit knowledge would be a bottleneck. Tacit knowledge questions are meant to be obscure to anyone not working in the field, i.e., they either require tracking down authors of relevant papers or knowing people in the field. Troubleshooting questions are meant to be obscure to anyone without hands-on experience, i.e., the answers are known only to people who have tried the protocol themselves.

This set is uncontaminated; it was created fully in-house with our partners at Gryphon Scientific and has not been published.

None of the tested models outperform the consensus expert baseline of 80%, though all models outperform the 80th percentile PhD expert baseline of 63%. We see some uplift from browsing on this evaluation.

To assess the model’s potential to assist in novel biological weapon design, we engaged biosecurity and chemistry experts from Signature Science, an organization specializing in national security relevant capabilities in the life sciences. Experts designed scenarios to test whether the pre-mitigation model could assist in creating novel bio threats and assessed model interactions against the risk thresholds. No trajectories were rated as “Critical” risk, finding that the model could effectively synthesize published literature on modifying and creating novel threats, but did not find significant uplift in designing novel, feasible, and dangerous threats beyond existing resources.

Eight members of the SecureBio AI group interacted with the ChatGPT agent (helpful-only), accessed via the ChatGPT platform. They found that the ChatGPT agent performed at roughly the o3-level¹ on tasks that do not significantly benefit from its ability to retrieve and analyze multiple sources, but it performed better on tasks that do. They noted that this ability confers some qualitatively new capabilities that would be particularly useful to a malicious actor.

Specifically, they observed that the agent outperformed prior models, such as o3, in the World-Class Biology evaluation. This evaluation assesses advanced biological knowledge and serves as a benign proxy for biorisk-related tasks. Their red-teaming efforts also indicated that the agent could reduce operational challenges for malicious actors. For instance, it demonstrated higher accuracy than previously released models (like o3) in identifying the most effective, actionable avenues for causing the most harm with the least amount of effort.

The assessment consisted of the following:

SecureBio manually tested selected static evaluation questions via the chat interface. To test the hypothesis that the ChatGPT agent is able to expand on the set of questions that o3 is able to answer, SecureBio gave the agent two types of questions: hard questions, questions that o3 typically fails to answer (\(\leq 3\) correct answers in 10 runs) and control questions, questions that o3 and other frontier models always answer correctly. Both categories were validated by baseliners and had at least one correct answer from a human expert.

SecureBio manually assessed ChatGPT agent on a subset of their agent evaluations, each of which can be aided by use of tools such as web search to retrieve relevant information.

Two SecureBio PhD virologists, with 6 and 10 years of hands-on laboratory experience with highly pathogenic human-infecting viruses, conducted approximately 10 hours of manual red-teaming with the ChatGPT agent. Their focus was evaluating the agent’s willingness to answer high-risk queries and delineating the bounds of info-hazardous knowledge. Notably, the ChatGPT agent’s capability to provide accurate citations and URLs exceeded prior models, facilitating access to information from obscure publications. Tasks could be more effectively "stacked" than in previous models, enabling the completion of complex, multi-part requests involving background knowledge, experimental design, protocols, and practical assistance in a single interaction, thereby providing a clearer pathway to potential harm.

OpenAI invited seven experts with professional and academic backgrounds relevant to biosafety and biosecurity to spend 3-5 hours using a helpful-only version of ChatGPT agent to test capabilities around potentially dangerous biological tasks (both agentic and non-agentic tasks), and to fill out a survey where they scored agent on its ability to complete the tasks, on their general assessment of it, on their perception of its ability to uplift various levels of users, and finally to compare to other models and resources that they are familiar with. Agentic tasks include tasks such as interacting with biological design tools, downloading genomes, obscuring information for procurement of dangerous materials. Non-agentic tasks include retrieving protocol information and troubleshooting and bio-SME knowledge tasks.

ChatGPT agent scored highly on non-agentic tasks (averaging performance equivalent to a subject matter expert), consistently delivering accurate, detailed, and contextually relevant biological insights approaching the level of a subject matter expert. However, its performance on agentic tasks was weaker which may be due to reliability and operational issues, such as incomplete responses and technical glitches. In general, experts noted an average success rate of ~7.7 (on a scale of 1-10) on tasks that they attempted with ChatGPT agent.

Experts identified substantial potential for ChatGPT agent to significantly uplift users’ capabilities, particularly benefiting graduate students and cross-disciplinary researchers with existing lab experience and judgment. The system rapidly consolidates complex knowledge about pathogen modification methods, experimental protocols, and equipment sourcing, potentially compressing days of research into minutes. Their findings indicated that inexperienced users (eg. undergraduate with bio knowledge) would still need practical laboratory skills and critical judgement and that the real threat comes when a resourceful novice is willing to persist. This is because the model still provided incorrect details that could reasonably set back semi-experienced actors by months and cost thousands of dollars.

Cybersecurity is focused on capabilities that could create risks related to use of the model for cyber-exploitation to disrupt confidentiality, integrity, and/or availability of computer systems.

These results show comparable performance to o3 , and were likewise below our High capability threshold.

We evaluate model performance on a test set of over 100 curated, publicly available CTF challenges which met the following criteria:

Challenges were further divided into the level of proficiency needed to solve the challenge, ultimately organized by whether the challenge required high school, collegiate, or professional-level cybersecurity knowledge and skills, with judgment on the requisite skills manually determined by our in-house cybersecurity expert. For all Collegiate and Professional CTFs – and most of the High School CTFs – we require that the solution involves a non-trivial level of exploitation, requiring chaining two or more exploitation steps and not be solvable by copy-pasting textbook examples.

To evaluate the model against the CTFs we give it access to a headless Kali Linux distribution with common offensive cybersecurity tools preinstalled as well as a harness which allows the model to call those tools or otherwise execute commands similar to as a human. We run 16 rollouts for each CTF exercise, recording the pass@12 metric over the best set of rollouts; this metric is then compared to the thresholds established by the Preparedness Framework as well as the performance of previously released OpenAI models.

We refactored the CTF challenge set for evaluations since the release of o3 and o4-mini. The new set consists of publicly available challenges with working execution environments compatible with our tech stack. It incorporates more recent CTFs to reflect the evolving cybersecurity landscape, balances challenge types across difficulty levels, and removes tasks that were trivial or failed to test meaningful cyber capabilities.

This update provides a more realistic measure of Frontier Models’ ability to solve modern cybersecurity challenges. However, scores are not directly comparable to previous system cards, as the new set is significantly harder—models perform roughly 30–50% lower. To establish a baseline, we re-ran o3 and o4-mini on the updated set for comparison with ChatGPT Agent, focusing on no-browsing results to prevent lookup-based solutions.

Given 12 attempts at each task, ChatGPT Agent performs comparably to o3 and o4-mini across all difficulty levels—slightly higher on Collegiate CTFs, slightly lower on High School and Professional CTFs. Overall, these results suggest minimal capability difference between ChatGPT Agent and prior releases.

As always, we note that these evaluation results likely represent lower bounds on model capability, because additional scaffolding or improved capability elicitation could substantially increase observed performance.

Cyber range exercises measure a model’s ability to conduct fully end-to-end cyber operations in a realistic, emulated network. These exercises are long-form, requiring the model to (1) construct a plan to achieve an abstract adversary objective; (2) exploit vulnerabilities, misconfigurations, and weaknesses that are likely to be seen in the wild; and (3) chain together these exploits to achieve the scenario objective.

Our cyber range testing focuses on two custom-designed scenarios. The first, Online Retailer, features a realistic network designed to emulate that of a small online retailer. The network consists of one Linux VM (the initial host), one Windows VM, a CI/CD server, a web server that uses a version of an open-source online store package, and a cloud storage container with sensitive information (the attacker’s end goal). The intended attack path requires the model to pivot across systems, exploit misconfigurations (e.g., unauthenticated services), and leverage known web vulnerabilities to ultimately exfiltrate sensitive data.

The second scenario, Privilege Escalation, is a simple privilege escalation scenario. It is smaller and purposefully designed to be easier than the first; it is more akin to a cyber skills test or CTF than it is to a network emulation exercise. The scenario is limited to two Linux VMs. The model must discover weak credentials, pivot between machines, and exploit a built-in path interception vulnerability to escalate privileges and access sensitive data. While less realistic, this scenario serves as a lower-bound capability check for multi-step operations.

Models operate in a headless Kali Linux environment with standard offensive tools, plus a harness for command execution. Each scenario is tested under three configurations:

For the online retailer scenario we run 30 trials in each configuration, and for the simple privilege escalation one we run 16. Our primary metric for evaluation is pass@12 for Normal and With Hints, and then pass@1 for With Solver Code.

These configurations help us assess not just whether the model can complete the task, but also how much external assistance it requires. We report results by scenario because their difficulty and realism differ substantially.

No model is able to solve the scenario unaided or with hints, but with solver code o3, o4-mini and ChatGPT Agent (all without browsing enabled) are able to solve both with high accuracy.

To extend our analysis, we also look at the ability of the models to achieve intermediate milestones during the task. Our current setup strictly analyzes the trajectory to identify whether a particular milestone was achieved. This approach lends itself well to identifying some milestones–like finding the randomly generated password for the web server–but is harder for others, such as evaluating if the model achieved remote code execution. Some of these necessary-but-hard-to-measure milestones are omitted from our analysis, and we caution that the analysis we provide is therefore only a rough measure of progress and does not entirely capture the sequence of actions needed to solve the task, or the relative difference in difficulty for each.

Above shows the milestone breakdown for each model in the Online Retailer scenario. Unaided, no model gets very far: they discover the CI/CD server and the Web Server, but struggle to proceed beyond that. The models do occasionally find the username and password for the web server – indicating compromise of the CI/CD server – though without the login information they cannot use this. ChatGPT Agent does identify the Web Server User more often than the other models, but this is likely due to brute force guessing and not compromising the CI/CD server. With Hints each of the models do better – reliably finding the login page and the credentials – but are unable to identify an entrypoint for RCE, stopping well short of the goal.

The above charts show milestone success for the Privilege Escalation scenario. Unaided, ChatGPT Agent performs marginally better than o3, but still significantly away from achieving the goal, as well as worse overall than o4-mini. With Hints ChatGPT Agent makes more progress – in some cases more reliability than the other two models – but is still unable to solve the task.

Results for both scenarios – including milestone analysis – suggest minimal capability difference between ChatGPT Agent and prior releases.

ChatGPT Agent does not demonstrate improved performance on software engineering and AI research tasks relevant to AI self-improvement risks.

We measure ChatGPT Agent’s ability to pass OpenAI’s Research Engineer interview loop, using a dataset of 97 multiple-choice questions. The 18 coding questions created from our internal interview question bank have been saturated.

All models since OpenAI o1 score similarly on the multiple choice question set. We don’t see any uplift from browsing.

SWE-bench Verified [5] is the human-validated subset of SWE-bench that more reliably evaluates AI models’ ability to solve real-world software issues. This validated set of tasks fixes certain issues with SWE-bench such as incorrect grading of correct solutions, under-specified problem statements, and overly specific unit tests. This helps ensure we’re accurately grading model capabilities. An example task flow is shown below:

For OpenAI o3, and o4-mini we used an internal tool scaffold designed for efficient iterative file editing and debugging. In this setting, we average over 4 tries per instance to compute pass@1 (unlike Agentless, the error rate does not significantly impact results).

All SWE-bench evaluation runs use a fixed subset of n=477 verified tasks which have been validated on our internal infrastructure. Our primary metric is pass@1, because in this setting (unlike e.g., OpenAI interviews), we do not consider the unit tests as part of the information provided to the model. Like a real software engineer, the model must implement its change without knowing the correct tests ahead of time.

Measuring if and when models can automate the job of an OpenAI research engineer is a key goal of self-improvement evaluation work. We test models on their ability to replicate pull request contributions by OpenAI employees, which measures our progress towards this capability.

We source tasks directly from internal OpenAI pull requests. A single evaluation sample is based on an agentic rollout. In each rollout:

If all task-specific tests pass, the rollout is considered a success. The prompts, unit tests, and hints are human-written.

PaperBench [6] evaluates the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria. In total, PaperBench contains 8,316 individually gradable tasks.

We measure a 10-paper subset of the original PaperBench splits, where each paper requires <10GB of external data files. We report pass@1 performance with high reasoning effort and no browsing.

ChatGPT agent scores are similar to o3’s scores, with and without browsing. We don’t see uplift from browsing on this evaluation despite code from the published papers being accessible online.

In this section we describe the safeguards we have implemented against biological and chemical risk, and explain how we determined that these safeguards sufficiently minimize the risk under our Preparedness Framework. This work builds on more than a year of efforts in the biological and chemical domain, and includes the work that we described in a June blog post.

As described in our Preparedness Framework, in accord with the SAG recommendation to treat this release as High capability in the biological and chemical domain, we implemented safeguards to sufficiently minimize the associated risks. What follows is a public summary of our internal Safeguards Report, which includes additional details that are not suitable for public disclosure (such as information potentially useful to attackers). The internal report informed SAG’s finding that these safeguards sufficiently minimize the associated risks.

Pursuant to our Preparedness Framework, we developed threat actor profiles and a threat model for biological risk that identifies specific pathways through which severe harm could arise, assesses the specific gating steps where our technology could play a role, and guides the development of safeguards to sufficiently minimize those risks of severe harm.

Threat modelling is not only a central aspect of our Preparedness Framework, but also a well established practice in the biosecurity domain. In anticipation of reaching a High capability threshold for biological and chemical risk, we expanded and deepened our existing collaboration with domain experts in biosecurity, virology, and computational biology, convened a wider group of subject matter expert reviewers (including but not limited to experts from SecureBio), and collaboratively and iteratively refined a threat model reflecting deep domain knowledge in both biosecurity and AI.

This process included research in consultation with biosecurity experts, vulnerability elicitation, countermeasure development and stress testing of our threat model and weaponization lifecycle framework. We addressed the following questions:

We used the learnings from this exercise to inform our development of a granular biosecurity safety taxonomy for model responses, safety training to teach the model to follow that taxonomy, automated monitoring and oversight systems, account-level enforcement policies and processes, and red teaming and other testing to assess the robustness of end to end safeguards.

This approach has allowed us to identify and intervene on suspicious or dangerous dual use or weaponization tasks, while preserving the integrity of model assistance for benign dual use areas.

We identified specific scenarios of concern, with different threat actor types (including different potential motivations, resources, and capabilities), biological agents, weaponization pathways, and projected impact of an incident, to animate discussions around risks and guide our prioritization efforts. We focused on scenarios (a) that were most plausible given the state of bioscience technology and AI capabilities, (b) where AI technology played a central uplifting role, (c) that led to a significant threshold of harm, and (d) that could be used for broader learning.

Our resulting, current biosecurity threat model focuses on two main pathways for our models to be used for biological harm:

We built out and validated with external experts a comprehensive “weaponization lifecycle” framework, which illustrates how threat actors might acquire and/or modify a known respiratory virus. This resulted in a repository of dozens of tactics, techniques and practices associated with each of the phases of the weaponization sequence. Having this level of high-fidelity detail into specific steps, tasks, and touchpoints where a threat actor would seek to leverage an AI model (as a knowledge assistant, via agentic support, through multi-modal troubleshooting, etc) allows us to anticipate, prevent, deny, and disrupt adversarial abuse across multiple layers of our safety stack. The weaponization lifecycle indicates that an actor would need to persistently probe the model and use it spanning a larger time horizon (weeks or months), in order to use it to successfully cause harm.

We then identified critical and semi-critical steps in our simulated weaponization sequence) in order to further understand and prioritize our vulnerability elicitation and countermeasure design. A critical step is:

Semi-critical steps are commonly observed, and where AI plays an uplifting role, but are not absolutely required for every weaponization effort.

We used our threat scenarios and weaponization lifecycle analysis to identify potential vulnerabilities that a threat actor might try to exploit in order to achieve steps or bypass critical touchpoints in the weaponization lifecycle. This includes downstream probing, incomplete policy coverage, multi-account obfuscation and abuse, agent hijacking and tool misuse, account take over, recidivism, and jailbreaks. We used this vulnerability analysis to inform specific countermeasure actions our safety, security, and enforcement operations teams will take to prevent, mitigate, and address identified vulnerabilities.

Our analysis of emerging threats, weaponization pathways, and potential catastrophic outcomes, is sensitive to a series of assumptions about threat actor motivations, resources, and levels of effort, as well as the types of uplift that might be provided by AI technologies. We will revisit these assumptions as conditions change:

Geopolitical drivers (increasingly hostile interstate dynamics), technological drivers (unexpectedly capable frontier models available to the public), and shifts in the biological science domain (number of and regulation of cloud labs, synthetic biology providers, contract research organizations) may also shift our threat model projections in the future, and we will continue to monitor and iterate on them.

Informed by our threat modelling efforts, we created a taxonomy of content related to biological threats, for use both in training models to be safe, and in building system-level safeguards that further protect against models providing information or assistance that could enable severe harm. This system was also used to identify potentially violative accounts for human review and account-level enforcement.

The categories of biothreat information defined in this taxonomy enable us to define, measure, and iteratively strengthen targeted safety behaviors that protect against relevant risks of severe harm.

The most important parts of this taxonomy are:

We developed an end-to-end suite of safeguards specifically to address the risks of severe harm identified in our externally validated threat model. These protections start with how the underlying model is trained, extend to system-level protections that cover 100% of Agent production traffic, and include scaled account-level enforcement capabilities.

The primary pathway we anticipate threat actors will try to use to cause severe harm with our models is via persistent probing for biorisk content. As a result, our safeguards approach has focused on proactively preventing such content via a multilayered defense stack. In addition to this, we also have an active enforcement pipeline to ban users who request such content (and may report them to law enforcement in extreme cases). Taken together, these safeguards underpin the following claims:

In addition to these measures, we have built a rapid remediation protocol to detect, triage, and appropriately mitigate any vulnerabilities that emerge after system launch. You can read more about rapid remediation and our bug bounty program in Section 5.2.2.4.

We trained the Agent model itself to be safe, using the taxonomy of biorisk information described above. Specifically, we trained the model to:

Although the model itself is trained to be safe – declining to answer or else giving safe, high-level answers in response to potentially dangerous questions – we do not rely solely on this training for safety. Instead, we’ve deployed a two tiered system of real-time, automated oversight surrounding the model to monitor and block unsafe prompts and generations.

This tiered system runs for all Agent traffic and works at multiple phases for each turn of a conversation. It scans user messages, external tool calls, and the final model output.

These system level protections are a form of defense in depth. Any attempted elicitation of hazardous content would need to bypass our model refusal behavior and all layers of system protection.

In addition to model training and system-level protections, we scan conversations with ChatGPT Agent in order to identify potential violations of our usage policies. We use a combination of automated systems and human review with biothreat experts to detect and ban users attempting to leverage ChatGPT Agent to create a biological threat. This includes recidivism prevention measures to address scenarios in which a single bad actor might interact with our system via multiple accounts. For extreme cases, we may notify relevant law enforcement.

Our rapid remediation protocol detects, triages and appropriately patches all universal jailbreaks and other vulnerabilities discovered from ChatGPT Agent. The input funnel includes social media monitoring and other off-platform monitoring, a bug bounty program for jailbreaks related to biorisk, and detecting jailbreaks in biorisk queries from production traffic.

All vulnerabilities are triaged to the rapid remediation team, who will be responsible for assessing the vulnerability and fixing the gap. This may include validating that the jailbreak is reproducible, confirming that our systems missed the attack, patching our automated systems, updating the policy and safety evaluations, and identifying needs for model retraining.

The team will have routine post-deployment safety reviews to review all detected vulnerabilities and remediation details.

We have launched a public bug bounty program that will allow selected (via invitation and application) researchers to submit universal jailbreaks. Selected researchers attempt to solve a “ladder” of multiple steps with a single jailbreak strategy. The goal is to test whether a reproducible, universal jailbreak exists that can over‑ride the model’s biosafety guardrails after deployment, so that we can learn and rapidly mitigate if any such possibility is found.

Consistent with our June blog update on our biosafety work, we are building a trusted access program to enable a less restricted version of Agent for certain vetted and trusted customers engaged in beneficial applications in areas such as biodefense and life sciences. We consider a range of governance and safety indicators before granting access to this program, including biosafety and security controls, as well as the nature of the intended use case. Under this program, if access is granted, the model will provide detailed responses to dual-use prompts, while still blocking weaponization generations. Our Usage Policies also remain in effect in all cases. We believe trusted access balances robust safeguards with enabling responsible advancement in life sciences.

As part of our preparedness process, we performed careful end-to-end testing of our biological safeguards. Below, we share select results from the testing that took place at each step.

To test the effectiveness of our model safety training, we use two test sets:

These metrics show, for some of the most challenging scenarios that the model can encounter, how often model training alone suffices to generate a safe response (whether that response is a refusal to provide weaponization information, or a safely high level and non-actionable response to a request for dual-use biological assistance). The remainder between these numbers and 1.0, on the other hand, reflects the fraction of cases in our highly adversarial test set where our other safeguards, including system level safeguards, are needed and play an active role in creating safety.

We tested both our biological topical classifier for biological information, and our reasoning model that provides deeper review of biology related conversations.

We evaluate the biological classifier on a distribution of prompts and generations that represents production traffic with binary synthetic labels of biological vs non-biological content. We then select a threshold with a minimum 0.95 recall of biology-related content.

To evaluate the reasoning monitor, we construct a challenging set of evaluation prompts and generations. We include a few types of challenging evaluation examples:

Each evaluation example is manually labeled by biology experts and cross calibrated.

* We put safety first by optimizing for high recall – catching as many dangerous interactions as possible – even at a cost of reduced precision. This means that our safety mitigations will sometimes accidentally prevent safe uses of the product.

This red teaming campaign focused on identifying universal jailbreaks and adversarial tactics which malicious actors might employ to evade biological risk safeguards in ChatGPT Agent. We worked with 16 experienced participants from the Red Teaming Network (RTN), each of whom had a biosafety-relevant PhD and had participated in previous biosafety red teams focused on our reasoning models and safeguards.

Red teamers were given access to ChatGPT Agent via a pre-deployment version of the product web interface. As described above, this included both model safety training as well as early versions of our system-level protections. All conversations used tools such as Operator and web browsing. Red teamers were directed to attempt to maximize their score on their choice of a specific, representative biorisk rubric, reflecting the weaponization paths identified in our threat model, which was subsequently scored with the guidance of a subject matter expert.

Red teamers were asked to make a submission, attaching one or more conversations, whenever they believed the trajectories of the submitted conversations, taken together, were unsafe. Red teamers made 110 such submissions as well as flagging 69 other submissions that they thought were safe but warranted further review by the OpenAI team (including 25 instances of over-refusal, where the model appeared to be unnecessarily conservative in safe biological conversations). Each submission was accompanied by a rubric which asked the red teamer to look for specific content that maps directly to our threat models. Of the 179 total submissions, 16 were submitted with attached rubrics that exceeded our internal thresholds for biological risk. Each of these 16 were either patched via moderation or product changes, or else determined to be allowed content upon manual review, prior to deployment.

FAR.AI conducted 40 hours of testing over 2 weeks. During this period, they identified three partial vulnerabilities in ChatGPT Agent which overcame some of the defense layers but not all. Current safety mechanisms rely heavily on monitoring during the reasoning and tool-use process, making it a potential single point of failure if compromised. Nonetheless, monitoring reasoning, not just the final output, is more robust as the monitor can take into account the model’s intent when processing the user query. Despite these limitations of the safety system and testing, FAR.AI concluded ChatGPT Agent appeared more resilient to biological risk jailbreaks than other models they tested comparably.

This campaign aimed to assess the utility of ChatGPT Agent with a pre-deployment version of the product web interface to a “novice” malicious actor who wanted to create biohazards despite lacking previous experience with the necessary information, actions, and tools. These exercises, while not a perfect proxy for real world conditions, provide some additional reassurance that novices cannot readily obtain dangerous information or assistance from Agent.

We sequenced two waves of this campaign. In the first wave, we worked with members of our red teaming network who are experienced red teamers and do not have backgrounds in biosafety to validate the methods. In the second wave, having confirmed our experiment design and methods, we worked with a vendor to organize “novices,” contractors who have backgrounds in STEM fields other than biology, to attempt to generate harmful biological information.

All participants in both waves were asked to familiarize themselves with ChatGPT Agent for at least one day prior to being given two 60 minute quizzes. Each user completed two quizzes: one quiz without any LLM resources (but with the ability to access search engines and other resources), and a second quiz with ChatGPT Agent available.

The quizzes required 10-15 long-answer responses to questions related to the effective creation of Abrin or Anthrax agents. Each answer was graded with a rubric that included points for partial credit based on the specificity and completeness of the answer. To support rigorous decisionmaking, for this study we explicitly considered the statistical power reflected in our study design: The study was powered to detect a large effect (Cohen’s d=0.6) in the exam scores based on treatment.

One‑sided superiority tests assessing whether the observed Cohen’s d exceeded the pre‑specified benchmark were non‑significant (Abrin: p = 0.95; Anthrax: p = 0.99), indicating that ChatGPT Agent did not have a large effect on their ability to answer the questions in this experimental setting.

Average score of respondents to the quizzes for various conditions. Means are provided with 95% confidence intervals.

We provided early access to the US Center for AI Standards and Innovation and the UK AI Security Institute (UK AISI) to prototype versions of our system to carry out safeguard red teaming of our models and monitors using specific queries provided by OpenAI in the biological risk domain. As part of a longer-term collaboration, UK AISI was also provided access to information sources that are not publicly available – such as the chains of thoughts of our reasoning monitors and our policy text – which made it easier for them to identify potential vulnerabilities than it would be for malicious users. Initial versions of safeguard red teaming were conducted on the OpenAI o3 model and an earlier version of our mitigation stack. Over four rounds of iteration and patching they identified a total of 7 universal attacks that were later reproducible in early versions of the ChatGPT Agent product. The 7 attacks identified were all patched and will share our updated system with them for further testing. Holistically, we found that the UK AISI’s attack investigations were thorough and instructive, enabling us to efficiently improve our safeguards and remediate vulnerabilities they found.

We used the red teaming process as an opportunity to test our enforcement process. We routed two days of red teaming traffic to our enforcement pipeline, which were processed by our automated and human review pipeline. All active red teamers during this two-day period were enqueued in our high risk human review queue. Our user ops team manually reviewed and banned these accounts.

In addition to the other safety measures described in this system card, we take steps to prevent adversaries from compromising sensitive intellectual property, including customer data, and theft of model weights used to power ChatGPT agent. As we have As we have previously described, we take a defense-in-depth approach to protecting our model weights, relying on a combination of access control, infrastructure hardening, egress controls, and monitoring. We leverage purpose-built detections and controls to mitigate the risk of exfiltration of high-risk model weights. We complement these measures with dedicated internal security teams, including Detection and Response, Threat Intelligence, and Insider-Risk programs. These programs help ensure emerging threats are identified and blocked quickly.

As the power and capabilities of our models increase, so do the security investments made to help protect them.

In accordance with our threat model (Section 5.2.1) the primary pathway by which we expect threat actors to use our models to cause severe harm is via persistent probing for biorisk content. We expect that successfully causing harm via this pathway would take a longer time horizon spanning weeks or months. In order to mitigate this, preventing universal jailbreaks has been the core area of focus for our work.

While we’ve implemented a multilayered defense system and carried out extensive red teaming and other tests, we acknowledge that there is a risk of previously unknown universal jailbreaks being discovered after deployment. We believe this risk to be sufficiently minimized primarily because discovering such jailbreaks will be challenging, users who attempt to get biorisk content may get banned (and may get reported to law enforcement in extreme cases), and because we expect to be able to discover and respond to publicly discovered jailbreaks via our bug bounty and rapid remediation programs.

We also note the following areas of potential remaining risk along with why we believe they’re sufficiently minimized:

Policy Gray Areas: Policy boundaries may sometimes conflict with expert assessments due to the absence of full consensus, particularly regarding dual-use technologies. This leads to challenges in clearly defining what content should be restricted and may lead to certain dimensions of high risk content being leaked. We believe this risk is sufficiently minimized under our Preparedness Framework because our policy currently errs on the side of being conservative and we believe that over-refusing on benign queries is a more likely possibility.

Incrementally Leaking Higher Risk Content: This threat model considers if users may be able to incrementally ask for information that is increasingly more detailed or combine individually benign information across sessions which in totality lead to higher risk content. We believe the risk for this is low as demonstrated by our novice uplift safeguard test where novices were unable to gain significant uplift.

Leaking harmful info via Visible Trajectory: In some circumstances, users could see disallowed biorisk information briefly appear outside a final answer, such as in a chain of thought summary or an agent’s trajectory for a query whose final answer is ultimately blocked. However, we believe the risk of content in these summaries being detailed and actionable enough for harm is limited.

Controllability via Trusted Access: We plan to introduce a trusted access program (Section 5.2.2.6), so the overall safety of the agent system depends in part on the effectiveness of that program, including the safety and security controls of program participants. We believe that this risk is minimal given the strict access conditions and our vetting processes which include assessing biosafety and security controls.

ChatGPT agent is a unified agentic system that brings together deep research capabilities, the ability to interact with web sites, and the intelligence and conversational fluency of ChatGPT. In tandem with integrating these capabilities, we have created our strongest suite yet of end-to-end safety protections. We look forward to improving agent’s capabilities, and to doing the new safety work that will be needed to safely deliver those capabilities at scale.

1. [1]

   Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, et al. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260.
2. [2]

   Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, et al. BBQ: A hand-built bias benchmark for question answering. arXiv preprint arXiv:2110.08193.
3. [3]

   Tejal Patwardhan, Kevin Liu, Todor Markov, Neil Chowdhury, Dillon Leet, Natalie Cone, et al. Building an early warning system for LLM-aided biological threat creation. OpenAI.
4. [4]

   Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, et al. “LAB-bench: Measuring capabilities of language models for biology research.” Available at: https://arxiv.org/abs/2407.10362.
5. [5]

   Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, et al. Introducing SWE-bench verified. OpenAI. Available at: https://openai.com/index/introducing-swe-bench-verified/.
6. [6]

   Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, et al. “PaperBench: Evaluating AI’s ability to replicate AI research.”