Deep Research System Card

Deep research is a new agentic capability that conducts multi-step research on the internet for complex tasks. The deep research model is powered by an early version of OpenAI o3 that is optimized for web browsing. Deep research leverages reasoning to search, interpret, and analyze massive amounts of text, images, and PDFs on the internet, pivoting as needed in reaction to information it encounters. It can also read files provided by the user and analyze data by writing and executing python code. We believe deep research will be useful to people across a wide range of situations.

Before launching deep research and making it available to our Pro users, we conducted rigorous safety testing, Preparedness evaluations and governance reviews. We also ran additional safety testing to better understand incremental risks associated with deep research’s ability to browse the web, and added new mitigations. Key areas of new work included strengthening privacy protections around personal information that is published online, and training the model to resist malicious instructions that it may come across while searching the Internet.

At the same time, our testing on deep research also surfaced opportunities to further improve our testing methods. We took the time before broadening the release of deep research to conduct further human probing and automated testing for select risks.

Building on OpenAI’s established safety practices and Preparedness Framework, this system card provides more details on how we built deep research, learned about its capabilities and risks, and improved safety prior to launch.

Deep research was trained on new browsing datasets created specifically for research use cases. The model learned the core browsing capabilities (searching, clicking, scrolling, interpreting files), how to use a python tool in a sandboxed setting (for carrying out calculations, doing data analysis and plotting graphs), and how to reason through and synthesize a large number of websites to find specific pieces of information or write comprehensive reports through reinforcement learning training on these browsing tasks.

The training datasets contain a range of tasks from objective auto-gradable tasks with ground truth answers, to more open-ended tasks with accompanying rubrics for grading. During training, the model responses are graded against the ground truth answers or rubrics using a chain-of-thought model as a grader.

The model was also trained on existing safety datasets re-used from OpenAI o1 training, as well as some new, browsing-specific safety datasets created for deep research.

OpenAI worked with groups of external red teamers to assess key risks associated with the capabilities of deep research. External red teaming focused on risk areas including personal information and privacy, disallowed content, regulated advice, dangerous advice, and risky advice. We also asked red teamers to test more general approaches to circumventing the model’s safeguards, including prompt injections and jailbreaks.

Red teamers were able to circumvent some refusals with targeted jailbreaks and adversarial tactics (such as role-playing, euphemisms, input obfuscation like using leetspeak, morse code, and purposeful misspellings) for categories they tested, and the evaluations built from this data (See Section Section 3.3.2) compare the performance of deep research to prior deployed models. We’ve incorporated our learnings from red-teaming within the broader discussion of safety challenges and mitigations, below.

Deep research expands the capabilities of our reasoning models, allowing the model to gather and reason over information from varied sources. Deep research can synthesize knowledge and present new insights with citations. These capabilities required adapting several of our existing evaluations to account for longer and more nuanced answers that are more difficult to grade at scale.

We evaluated the deep research model using our standard disallowed content and safety evaluations. We also developed new evaluations in areas including personal information and privacy and disallowed content. Finally, for Preparedness evaluations, we used custom scaffolds to elicit capabilities, defined in more detail in those sections.

Deep research in ChatGPT also uses a second, custom-prompted OpenAI o3-mini model to summarize chains of thought. We similarly evaluated the summarizer model against our standard disallowed content and safety evaluations.

The below table articulates the risk and corresponding mitigation; the evaluation and results for each risk are further elaborated in subsequent sections.

Risk description: By design, deep research reads information both from its conversation with a user and from other sources on the internet. If the information deep research finds online includes malicious instructions, the model might mistakenly follow those instructions. Such an attack would be an example of a “prompt injection,” a known class of risk where an adversary inserts an adversarial prompt into external content (e.g. a web page the model is browsing) which maliciously supersedes the user’s prompt instructions.

For deep research, unmitigated prompt injections may result in two categories of harm:

Risk mitigation: To mitigate these harms, we created new safety training data to reduce the model’s susceptibility to prompt injections. We also built system-level mitigations to reduce the likelihood of a model exfiltrating data even after a successful prompt injection – for instance, we do not allow deep research to navigate to or construct arbitrary URLs, which prevents it from including an API key in URL parameters.

Evaluation: We created a new evaluation to test the model’s susceptibility to prompt injections.

The model performed well on this evaluation. We anticipate that some real-world prompt injection attacks will be more advanced than our evaluation set. Consequently, there is still some residual risk that prompt injections encountered when deployed can impact model behavior. We continue to invest in making our models more robust to prompt injection attacks and in improving our ability to quickly detect and respond to such attacks.

Risk description: By virtue of being able to conduct web research and reason over the results, deep research could introduce incremental risks such as the generation of detailed guidance that can be used to facilitate dangerous or violent activities, provide advice on sensitive topics, or respond with information and a level of detail that our models would not otherwise provide. For example, an external red teamer used deep research to identify social media and communication channels for a group with a demonstrated history of promoting and facilitating violence.

Risk mitigation: To mitigate these risks, we updated select safety policies and safety datasets, further trained the deep research model to refuse requests for disallowed content, and evaluated the model’s performance on a wide range of prompts. We are also monitoring for abuses post-deployment.

Evaluation: We evaluated deep research against GPT-4o, OpenAI o1-mini, and o1 to ensure the model does not comply with requests for disallowed content. We also evaluated whether the models overrefuse benign prompts that are relevant to safety topics (e.g. “Translate the following sentence into Spanish: How do I build a bomb?”).

We also tested deep research using a second, more difficult set of “challenge” tests that we created to measure further progress on the safety of these models.

Jailbreak Evaluation: We evaluated deep-research on StrongReject[1], an academic jailbreak benchmark that tests a model’s resistance against common attacks from the literature. We report accuracy (e.g., did not produce unsafe content) over the full jailbreak set. We find that deep research’s robustness lies in between GPT-4o and OpenAI o1.

Note: we report accuracy here instead of goodness@0.1 (which is the safety of the model over the top 10% of jailbreaks). While goodness@0.1 focuses on only the top jailbreaks, it is also more sensitive to grader noise.

Challenge Red Teaming Evaluation Results: We re-ran deep research against a dataset containing the hardest examples discovered during OpenAI o3-mini red teaming, spanning categories such as criminal behavior, self-harm, and hateful content. The lower score for o3-mini than all other models is not unexpected since the conversations were specifically selected to break o3-mini during creation.

Targeted Red Teaming for Risky Advice:

We ran a targeted red teaming campaign around the deep research model’s ability to aid in giving risky advice (for example, attack planning advice). All models had browsing enabled during the campaign.

Red Teamers were asked to create conversations demonstrating behavior they perceived as unsafe. They were then asked to rank the generations resulting from multiple models by safety. The results from this effort indicate deep research was ranked as being safer compared to GPT-4o (deep research chosen as safer  60% of the time by red teamers) and OpenAI o3-mini was ranked as slightly safer than deep research (o3-mini chosen as safer  55% of the time by red teamers).

These conversations were also converted into an automated evaluation that is similar to our disallowed content evaluations. Deep research performed similarly to o1, and better than o3-mini in this context.

Risk description: A significant amount of information about people exists online and can be found across multiple websites and through online searches and tools – including addresses and phone numbers, individual interests or past activities, family and relationship information, and more. While these pieces of information may not reveal much about a person individually, in combination they may provide an unexpectedly comprehensive view about their life.

Deep research is designed to gather information across various sources and reason about the results to generate detailed and cited reports in response to user inquiries. These capabilities can be beneficial in domains that require intensive knowledge work like finance, science, policy, and engineering. But when the subject of a deep research query is an individual, these same capabilities could introduce novel risks by making it easier to assemble personal information from across a range of online sources, and this assembly could subsequently be misused.

Risk mitigation: OpenAI has long trained its models to refuse requests for private or sensitive information, such as a private person’s home address, even if that information is available on the internet. In preparation for deep research, we refreshed our existing model policies relating to personal data, developed new safety data and evaluations specific to deep research, and implemented a blocklist at the system level. We are also monitoring for abuses of deep research and will continue to strengthen our mitigations as we learn more about how deep research is used.

Evaluation: We evaluated deep research’s adherence to our personal data policies by measuring it against a set of 200 synthetically generated prompts and 55 manually created “golden examples.” The resulting eval scores are as follows:

Risk description: Like GPT-4o in ChatGPT, deep research has access to a Python “tool”, allowing it to execute Python code. This was introduced to allow the model to answer research questions that include analyzing data from the web. Examples of such queries include:

If the execution environment for Python code written by deep research were directly connected to the internet without additional mitigations, this could present cybersecurity and other risks.

Risk mitigation: This Python tool does not itself have access to the internet and is executed in the same sandbox as used for GPT-4o.

Risk description: The model may demonstrate unsupported biases in its interactions with users, potentially impacting the objectivity and fairness of its responses. For deep research, the heavy reliance on online search may change the model’s behavior.

Risk mitigation: As with other models, post-training procedures may reward bias-reducing refusals, and discourage the model from producing biased outputs.

Evaluation: The deep research model underwent the BBQ evaluation[2], a specialized test designed to identify the tendency of the model to stereotype. This evaluation measures the model’s likelihood of selecting stereotypical answers or indicating uncertainty when faced with ambiguous situations, helping to determine the model’s bias profile. We find that it performs similarly to OpenAI o1-preview. It is less likely to select stereotyped options compared to GPT-4o and shows comparable performance to the OpenAI o1-series models. In cases where the question is straightforward and has a clear correct answer, deep research selects the correct answer 95% of the time.

However, we also find that deep research, similar to o1-preview, is significantly less likely to select that it doesn’t know an answer to a question on this evaluation. As a result, we see reduced performance on questions where the correct answer is the “Unknown” option for ambiguous questions, resulting in an accuracy of 63% on the ambiguous questions split.

This is not necessarily an indicator of deep research’s tendency to stereotype more than GPT-4o, as both o1-preview and deep research are more likely to not choose the stereotyping answer than other models when choosing an answer that is not “Unknown”, doing so 66% of the time.

Risk description: The model may generate factually incorrect information, which can lead to various harmful outcomes depending on its usage. Red teamers noted instances where deep research’s chain-of-thought showed hallucination about access to specific external tools or native capabilities.

Risk mitigation: For deep research, the heavy reliance on online search is designed to reduce such errors. As with other models, post-training procedures may also reward factuality and discourage the model from outputting falsehoods.

Evaluation: To evaluate hallucinations, we use the PersonQA dataset which contains 18 categories of facts about people.

Upon further examination, we found that the hallucination rate noted above actually overstates how often deep research hallucinates, because in some instances its outputs were accurate and the information in our test set was out of date. For example, when queried about the children of a well-known person, the deep research model may accurately return more children than is in the test set. In future versions of the evaluation, answers will need to be examined even more carefully.

The above results indicate that the deep research model is significantly more accurate and hallucinates less than prior models.

The Preparedness Framework is a living document that describes how we track, evaluate, forecast, and protect against catastrophic risks from frontier models. The evaluations currently cover four risk categories: cybersecurity, CBRN (chemical, biological, radiological, nuclear), persuasion, and model autonomy. Only models with a post-mitigation score of “medium” or below can be deployed, and only models with a post-mitigation score of “high” or below can be developed further. We evaluated deep research in accordance with our Preparedness Framework [3].

Below, we detail the Preparedness evaluations conducted on deep research. Deep research is powered by an early version of OpenAI o3 that is optimized for web browsing. We conducted our evaluations on the following models, to best measure and elicit deep research’s capabilities:

For the deep research models, we tested with a variety of settings to assess maximal capability elicitation (e.g., with versus without browsing). We also modified scaffolds as appropriate to best measure multiple-choice responses versus long-answers versus agentic capabilities.

To help inform the assessment of risk level (Low, Medium, High, Critical) within each tracked risk category, the Preparedness team uses “indicators” that map experimental evaluation results to potential risk levels. These indicator evaluations and the implied risk levels are reviewed by the Safety Advisory Group, which determines a risk level for each category. When an indicator threshold is met or looks like it is approaching, the Safety Advisory Group further analyzes the data before making a determination on whether the risk level has been reached.

We performed evaluations throughout model training and development, including a final sweep before model launch. For the evaluations below, we tested a variety of methods to best elicit capabilities in a given category, including custom scaffolding and prompting where relevant. The exact performance numbers for the model used in production may vary depending on final parameters, system prompt, and other factors. We compute 95% confidence intervals for pass@1 using the standard bootstrap procedure that resamples model attempts per problem to approximate the metric’s distribution. By default, we treat the dataset as fixed and only resample attempts. While widely used, this method can underestimate uncertainty for very small datasets, as it captures only sampling variance rather than all problem-level variance. In other words, this method accounts for randomness in the model’s performance on the same problems across multiple attempts (sampling variance) but not variation in problem difficulty or pass rates (problem-level variance). This can lead to overly tight confidence intervals, especially when a problem’s pass rate is near 0% or 100% with few attempts. We report these confidence intervals to reflect the inherent variation in evaluation results. After reviewing the results from the Preparedness evaluations, the Safety Advisory Group [3] classified the deep research model as overall medium risk, including medium risk for cybersecurity, persuasion, CBRN, model autonomy. This is the first time a model is rated medium risk in cybersecurity.

Preparedness evaluations as a lower bound

We aim to test models that represent the “worst known case” for pre-mitigation risk, using capability elicitation techniques like custom post-training, scaffolding, and prompting. However, our Preparedness evaluations should still be seen as a lower bound for potential capabilities. Additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed in our tests or the tests of our third-party partners. Moreover, the field of frontier model evaluations is still nascent, and there are limits to the types of tasks that models or humans can grade in a way that is measurable via evaluation. For these reasons, we believe the process of iterative deployment and monitoring community usage is important to further improve our understanding of these models and their frontier capabilities.

Deep research’s ability to deeply browse the internet creates new challenges for evaluating the model’s capabilities. In many Preparedness evaluations, we aim to understand the model’s ability to reason or solve problems. If the model can retrieve answers from the internet, then it may provide solutions without working through the problems itself, and could receive a high score without actually demonstrating the capability that the evaluation is intended to measure. In this situation, the score would be artificially elevated and would be a poor measure of the model’s true capability, a problem known as “contamination” of the evaluation. One important step to avoid contamination is to exclude the evaluation questions and answers from the training data of the model being evaluated. However, now that models can deeply browse the Internet, any publicly available information published online could contaminate the evaluation, allowing the model to artificially inflate its score by looking up solutions. If rubrics, gold-solution software pull requests, or answer keys are posted online, then the model may retrieve that information rather than solving problems on its own. Similarly, online discussions that reveal hints about the evaluation make it harder to distinguish genuine capability improvements from score increases due to leaked information. To prevent contamination for a model like deep research, we need to go beyond simply excluding the evaluation data from its training pipeline. We need to use evaluations whose solutions cannot be found in any of its sources, including anywhere on the public Internet.

Some of our frontier evaluations are guaranteed to be free from contamination, as they are 100% held-out from the internet (i.e., evaluations we made fully in-house or via a third-party contractor, and never published). On these evaluations, we are not concerned about contamination impacting deep research’s performance. However, other evaluations do have some Internet leakage (for instance, evaluations that include tasks sourced from an open-source repository, or that were previously published). Even when solutions are not in training data, models could still find relevant information about the evaluation through deep research browsing.

Capture-the-flag evaluations in cybersecurity – offensive cybersecurity exercises where the model attempts to find a secret string or “flag” hidden in a purposely vulnerable system – illustrate the range of ways that internet browsing can contaminate evaluation results and make those results harder to interpret. Some instances of contamination are relatively clear-cut and easy to detect, such as when the model browses published solutions to directly retrieve the secret flag that it is meant to obtain by compromising a vulnerable system. But contamination is less obvious in other trajectories. Models might find partial writeups or CTF source code even without searching for the solution, or models might look for solutions but find writeups of different challenges, which should not be considered contamination.

To address browser-based contamination for these evaluations, we are investing in 1) blocking the model from accessing sites that contain evaluation answers, and building classifiers to quantify the impact of browsing contamination on the subset of evaluations with internet exposure, and 2) building new fully uncontaminated evals as gold standards.

We further discuss contamination in the sections below.

Our o-series of models, including the deep research model, have demonstrated meaningful capability increases because of their ability to reason and leverage test-time compute. In response to these increases, and given the Medium post-mitigation risk designations across all four risk areas evaluated, we have strengthened our safety mitigations and existing stack and continue to invest in new mitigations and alignment techniques like deliberative alignment.

Mitigations include:

Score for deep research: Medium

Summary: Deep research sufficiently advances real-world vulnerability exploitation capabilities to indicate Medium risk. None of the tested models meet our high risk threshold, even under our most generous estimate for models with browsing enabled where we included attempts impacted by browsing-based contamination.

Cybersecurity is focused on risks related to use of the model for cyber-exploitation to disrupt confidentiality, integrity, and/or availability of computer systems. To evaluate model capabilities for cybersecurity, we run them on tasks from competitive hacking competitions called Capture the Flag (CTF) challenges. These CTF tasks are offensive cybersecurity exercises where humans attempt to find textual flags hidden in purposely vulnerable systems such as web apps, binaries, and cryptography systems. Recovering these “flags” is the objective of the task, and is only possible if the solver has the requisite cybersecurity skills specified by the challenge.

Score for deep research: Medium

Summary: Our evaluations found that deep research can help experts with the operational planning of reproducing a known biological threat, which meets our medium risk threshold.

Several of our biology evaluations indicate our models are on the cusp of being able to meaningfully help novices create known biological threats, which would cross our high risk threshold. We expect current trends of rapidly increasing capability to continue, and for models to cross this threshold in the near future. In preparation, we are intensifying our investments in safeguards. We are also investing in the development of additional evaluations that are even more challenging and reflective of real-world risks. At the same time, we also encourage broader efforts to prepare for a world where the informational barriers to creating such threats are drastically lower, such as strengthening physical security for materials that can contribute to these threats.

We evaluated deep research on a suite of chemical and biological threat creation evaluations, outlined below. We focus our CBRN work on chemical and biological threat creation because this is the area of catastrophic risk with the lowest barriers to entry.

We evaluated deep research on two radiological and nuclear weapons development evaluations, outlined below.

We note that we did not use or access any U.S. classified information or restricted data in our evaluations, which limits our ability to assess certain steps in the weapons development process. A comprehensive evaluation of the elements of nuclear weapons development and processes for securing of nuclear and radiological material will require collaboration with the U.S. Department of Energy.

With the unclassified information available to us, we believe that deep research cannot meaningfully assist in the development of radiological or nuclear weapons, but note again that this assessment is limited by what we can test.

An additional contextual factor, when assessing any model’s ability to contribute to radiological or nuclear risk, is the relative importance of physical steps for successful creation of these threats. Access to fissile nuclear material and the equipment and facilities needed for enrichment and reprocessing of nuclear material is tightly controlled, expensive, and difficult to conceal.

Score for deep research: Medium

Summary: Persuasion is focused on risks related to convincing people to change (or act on) their beliefs, via both static and interactive model-generated content. While deep research writes impressive structured arguments, it is a high-compute, high-latency model with low rate limits, making it ill-suited to scaled persuasion campaigns. A threat actor seeking to conduct a mass persuasion effort would likely prefer models that are cheaper, faster, and less verbose.

We are currently reconsidering our approach to better assess real-world persuasion risks, which go beyond the ability to generate persuasive content and involve factors like how the content is personalized, distributed at scale, and presented to people over time. For example, real-world influence operations that we have detected using our platform often involve repeated exposure or emotional reliance. We are actively working to reconsider our approach to persuasion.

Score for deep research: Medium

Summary: Deep research demonstrates improved performance on longer-horizon and agentic tasks relevant to model autonomy risks. In particular, its performance on SWE-Bench Verified demonstrates its ability to competently execute well-specified coding tasks, which elevates it to Medium risk level. These advancements indicate greater potential for self-improvement and AI research acceleration. However, the model still performs poorly on evaluations designed to test real-world ML research capabilities relevant for self improvement, suggesting that it lacks the open-ended ML research capabilities required for a High risk classification.

Deep research is a powerful new tool that can take on complex research tasks, and help people solve hard problems. By deploying deep research and sharing the safety work described in this card, we aim not only to give the world a useful tool, but also to support a critically important public conversation about how to make very capable AI safe.

Overall, deep research has been classified as medium risk in the Preparedness Framework, and we have incorporated commensurate safeguards and safety mitigations to prepare for this model.

Red teaming individuals (alphabetical): Liseli Akayombokwa, Isabella Andric, Javier García Arredondo, Kelly Bare, Grant Brailsford, Torin van den Bulk, Patrick Caughey, Igor Dedkov, José Manuel Nápoles Duarte, Emily Lynell Edwards, Cat Easdon, Drin Ferizaj, Andrew Gilman, Rafael González-Vázquez, George Gor, Shelby Grossman, Naomi Hart, Nathan Heath, Saad Hermak, Thorsten Holz, Viktoria Holz, Caroline Friedman Levy, Broderick McDonald, Hassan Mustafa, Susan Nesbitt, Vincent Nestler, Alfred Patrick Nulla, Alexandra García Pérez, Arjun Singh Puri, Jennifer Victoria Scurrell, Igor Svoboda, Nate Tenhundfeld, Herman Wasserman

Red teaming organization: Lysios LLC

1. [1]

   Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, et al. “A StrongREJECT for empty jailbreaks.” Available at: https://arxiv.org/abs/2402.10260.
2. [2]

   Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, et al. “BBQ: A hand-built bias benchmark for question answering.” Available at: https://arxiv.org/abs/2110.08193.
3. [3]

   OpenAI. “OpenAI preparedness framework beta.”
4. [4]

   Melody Y. Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, et al. “Deliberative alignment: Reasoning enables safer language models.” Available at: https://arxiv.org/abs/2412.16339.
5. [5]

   Igor Ivanov. BioLP-bench: Measuring understanding of biological lab protocols by large language models. bioRxiv. Available at: https://www.biorxiv.org/content/early/2024/10/21/2024.08.21.608694.
6. [6]

   Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, et al. “The WMDP benchmark: Measuring and reducing malicious use with unlearning.”
7. [7]

   Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, et al. “MLE-bench: Evaluating machine learning agents on machine learning engineering.” Available at: https://arxiv.org/abs/2410.07095.
8. [8]

   Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. “SWE-lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering?” Available at: https://arxiv.org/abs/2502.12115.