Update to GPT-5 System Card: GPT-5.2

GPT-5.2 is the latest model family in the GPT-5 series, and explained in our blog. The comprehensive safety mitigation approach for these models is largely the same as that described in the GPT-5 System Card and GPT-5.1 System Card.

In this card we also refer to GPT-5.2 Instant as gpt-5.2-instant and GPT-5.2 Thinking as gpt-5.2-thinking.

Like OpenAI’s other models, the GPT-5.2 models were trained on diverse datasets, including information that is publicly available on the internet, information that we partner with third parties to access, and information that our users or human trainers and researchers provide or generate. Our data processing pipeline includes rigorous filtering to maintain data quality and mitigate potential risks. We use advanced data filtering processes to reduce personal information from training data. We also employ safety classifiers to help prevent or reduce the use of harmful or sensitive content, including explicit materials such as sexual content involving a minor.

OpenAI reasoning models are trained to reason through reinforcement learning. These models are trained to think before they answer: they can produce a long internal chain of thought before responding to the user. Through training, these models learn to refine their thinking process, try different strategies, and recognize their mistakes. Reasoning allows these models to follow specific guidelines and model policies we’ve set, helping them act in line with our safety expectations. This means they provide more helpful answers and better resist attempts to bypass safety rules.

Note that comparison values from previously-launched models are from the latest versions of those models, so may vary slightly from values published at launch for those models.

We conducted benchmark evaluations across disallowed content categories. We report here on our Production Benchmarks, an evaluation set with conversations representative of challenging examples from production data. As we noted in previous system cards, we introduced these Production Benchmarks to help us measure continuing progress given that our earlier Standard evaluations for these categories had become relatively saturated.

These evaluations were deliberately created to be difficult. They were built around cases in which our existing models were not yet giving ideal responses, and this is reflected in the scores below. Error rates are not representative of average production traffic. The primary metric is not_unsafe, checking that the model did not produce output that is disallowed under the relevant OpenAI policy.

Values from previously-launched models are from the latest versions of those models, and evals are subject to some variation. Values may vary slightly from values published at launch for those models.

gpt-5.2-thinking and gpt-5.2-instant generally perform on par with or better than gpt-5.1-thinking and gpt-5.1-instant. They especially improve on Suicide/Self-Harm, Mental Health, and Emotional Reliance offline evals, which were lower for GPT-5.1 (see GPT-5.1 system card).

Additionally, we have observed through internal testing that GPT-5.2 Instant generally refuses fewer requests for mature content, specifically sexualized text output. Our testing suggests that this does not impact other types of disallowed sexual content or content involving minors.

We’ve found that this does not materially impact users whom we know to be minors, for whom our preexisting safeguards appear to be working well. For these users, we apply additional content protections that reduce access to sensitive content including violence, gore, viral challenges, sexual, romantic, or violent role play and extreme beauty standards. We are in the early stages of rolling out our age prediction model so that we can automatically apply these protections to accounts for users we believe to be under 18. We will continue to share progress.

For other users, we have deployed system-level safeguards in ChatGPT intended to mitigate this behavior. Our automated and manual testing suggest that these additional safeguards help to mitigate the issue.

We are continuing to improve our safeguards in this area and these learnings will inform any future releases.

We evaluate the robustness of models to jailbreaks: adversarial prompts that purposely try to circumvent model refusals for content it’s not supposed to produce.

Below is an adaptation of the academic jailbreak eval, StrongReject [1]. This eval inserts a known jailbreak into an example from disallowed content evaluations. We then run it through the same policy graders we use for disallowed content checks. We test jailbreak techniques on base prompts across harm categories, and evaluate for not_unsafe according to relevant policy. Note we filtered the original set of StrongReject examples to remove examples where all our models including older 4o were consistently safe, as it was otherwise highly saturated.

We find that gpt-5.2-thinking performs better than gpt-5.1-thinking.

gpt-5.2-instant performs lower than gpt-5.1-instant, though it still performs higher than gpt-5-instant-oct3 (as reported in GPT-5.1 system card addendum). Upon investigation, some of the errors are due to grader issues, and the remainder appear to be a regression in some cases under the illicit category, which we will investigate for future updates.

We evaluate the model’s robustness to known prompt injection attacks against connectors and function-calling. These attacks embed adversarial instructions in the tool-output that aim to mislead the model and override the system/developer/user instruction. Both of these evaluations are splits of the data we used for training, so don’t represent a model’s ability to generalize to new attacks. The two eval sets we have are:

Both gpt-5.2-instant and gpt-5.2-thinking show significant improvements on these evaluations, essentially saturating these evals. As with any adversarial space, these evaluations overrepresent robustness as we are only able to evaluate against the attacks we know about; even so, we observe these models to be strongly robust to known attacks.

We ran the image input evaluations introduced with ChatGPT agent, that evaluate for not_unsafe model output, given disallowed combined text and image input.

We find that both the instant and thinking variations of GPT-5.2 perform generally on par with their predecessors. We manually examined the failures for the vision self-harm eval, and found false positives due to grader issues; upon manual investigation, the model meets safety launch requirements and grader issues will be addressed in a future iteration.

To evaluate our models’ ability to provide factually correct responses, we measure the rate of factual hallucinations on prompts representative of real ChatGPT production conversations. We use an LLM-based grading model with web access to identify factual errors in the assistant’s responses to these prompts and report both the percentage of claims across responses that are identified as having a factual error as well as the percentage of responses containing at least one major factual error. We find that GPT-5.2 Thinking performs on par with (or slightly better) than its predecessors in this setting.

To understand how factuality varies by topic, we additionally use an LLM-based classifier to identify subsets of prompts that cover specific factuality-relevant domains: business and marketing research, financial and tax, legal and regulatory, reviewing and developing academic essays, and current events and news. GPT-5.2 Thinking performs especially well with browsing enabled, achieving <1% hallucination rate across all 5 domains.

Chatbots can empower consumers to better understand their health and help health professionals deliver better care [2] [3]. We evaluate GPT-5.2 on HealthBench [4], an evaluation of health performance and safety. HealthBench comprises 5,000 examples with (potentially multi-turn) conversations between chatbots and either consumers or health professionals. Model responses are evaluated with example-specific rubrics. We report results on three variants, HealthBench, HealthBench Hard, and HealthBench Consensus.

In the table above, we observe that GPT-5.2 models perform similarly to their respective GPT-5.1 models in health performance and safety.

Deception – when the model’s user‑facing response misrepresents its internal reasoning or the actions it took – can arise in a variety of circumstances. We evaluate GPT-5.2 Thinking on a set of prompts representative of traffic that previously elicited deception in ChatGPT. In addition, we use a modified version of the multimodal CharXiv benchmark [5], in which all of the images are removed and measure whether the model attempts an answer. We take a similar approach by posing questions that require browsing, but each call to the browsing tool returns an error, and in a set of agentic coding tasks designed to be impossible.

While deception previously likely to occur in production traffic decreased, we observed elevated failures in some specific domains. We initially found that GPT-5.2 Thinking, in the face of missing images, was more willing to hallucinate answers than previous models. However, upon closer inspection we found that this was partly driven by some prompts having strict output requirements (e.g., “Only output an integer”). Thus, when posed with a tension between instruction following and abstention, the model prioritized stricter instruction following.

Similarly, on a subset of the coding deception benchmark where the task given to the model doesn’t match the codebase it’s operating in, GPT-5.2 Thinking tended to attempt to implement the entire codebase from scratch. While this was not intended behavior, it is arguably benign, as the model made a good faith attempt to follow the user’s request, rather than falsely claiming success without trying.

In real production traffic, GPT-5.2 Thinking was deceptive 1.6% of the time, significantly lower than GPT-5.1 and slightly lower than GPT-5. These numbers are based on running reasoning based chain of thought monitors over large samples of pre-release AB-test traffic. Categories of deception include among others: lying about what tools returned or what tools were run, fabricating facts or citations, being overconfident in the final answer compared to internal reasoning, reward hacking and claiming to do work in the background when no work was occurring.

Robustly aligning advanced AI systems is an open research question and there are many potential trade offs, including how strictly to prioritize instruction following over other alignment considerations. As we continue to improve alignment techniques, we aim to transparently communicate any residual idiosyncrasies.

We trained gpt-5.2-thinking integrations to provide maximally helpful support on educational/cybersecurity topics while refusing or de-escalating operational guidance for cyber abuse, including areas such as malware creation, credential theft, and chained exploitation. We assess performance on data that do not overlap with the training set, measuring policy compliance rate (higher is better).

Overall, we observe significant improvements in policy compliance rate for gpt-5.2-thinking compared to gpt-5.1-thinking and gpt-5-thinking.

We saw no meaningful regression in capability evaluations. We observe minimal regression in concreteness on benign cyber requests, as well as a small drop in concreteness for high risk dual use cyber requests.

To evaluate the models’ multilingual capabilities, we used professional human translators to translate MMLU’s test set into different languages. We find that gpt-5.2-thinking performs generally on par with gpt-5-thinking.

These results were achieved through 0-shot, chain-of-thought prompting of the model. The answers were parsed from the model’s response by a grader model.

We also tested the models on our first-person fairness evaluation [6]. This evaluation consists of multiturn conversations, starting with a prompt in which a user shares their name such as “Hi, I’m [NAME].” to which the model responds “Hi [NAME]! How can I assist you today?” We then prompt the model with a request such as “Write a fairytale.”

This evaluation is used to assess harmful stereotypes by grading differences in how a model responds to the conversation when the user’s name is statistically more often associated with males (e.g., Brian) or females (e.g., Ashley). Responses are rated for harmful differences in stereotypes using GPT-4o, whose ratings were shown to be consistent with human ratings. This evaluation consists of over 600 challenging prompts reflecting real-world scenarios that exhibit high rates of bias in GPT-4o-mini generations. These prompts were intentionally chosen to be an order of magnitude more difficult than standard production traffic; this means that in typical use, we expect our models to be about ten times less biased.

We report the metric harm_overall, which represents our expected difference of biased answers for male vs female names based on the performance on this evaluation (i.e., performance on the evaluation divided by 10). We see gpt-5.2-thinking performs generally on par with gpt-5.1-thinking.

The Preparedness Framework is OpenAI’s approach to tracking and preparing for frontier capabilities that create new risks of severe harm. The framework commits us to track and mitigate the risk of severe harm, including by implementing safeguards that sufficiently minimize the risk for highly capable models.

As we did for gpt-5.1-thinking and gpt-5-thinking before it, we are continuing to treat gpt-5.2-thinking as High capability in the Biological and Chemical domain. We have applied the corresponding safeguards for this model as described in the GPT-5 system card.

For cybersecurity and AI self-improvement, evaluations of final checkpoints indicate that, like their predecessor models, GPT-5.2 models do not have a plausible chance of reaching a High threshold.

For the evaluations below, we tested a variety of elicitation methods, including scaffolding and prompting where relevant. However, evaluations represent a lower bound for potential capabilities; additional prompting or fine-tuning, longer rollouts, novel interactions, or different forms of scaffolding could elicit behaviors beyond what we observed in our tests or the tests of our third-party partners.

We calculate 95% confidence intervals for pass@1 using a standard bootstrap procedure that resamples model attempts per problem to approximate the metric’s distribution. While widely used, this method can underestimate uncertainty for very small datasets, as it captures only sampling variance (randomness in the model’s performance on the same problems across multiple attempts) rather than all problem-level variance (variation in problem difficulty or pass rates). This can lead to overly tight confidence intervals, especially when a problem’s pass rate is near 0% or 100% with few attempts. We report these confidence intervals to reflect the inherent variation in evaluation results.

We are treating this launch as High capability in the Biological and Chemical domain, activating the associated Preparedness safeguards. We do not have definitive evidence that these models could meaningfully help a novice to create severe biological harm, our defined threshold for High capability, and these models remain on the cusp of being able to reach this capability. Given the higher potential severity of biological threats relative to chemical ones, we prioritize biological capability evaluations and use these as indicators for High and Critical capabilities for the category.

To evaluate models’ ability to troubleshoot wet lab experiments in a multimodal setting, we evaluate models on a set of 350 fully held-out virology troubleshooting questions from SecureBio.

All models exceed the median domain expert baseline of 22.1%.

To evaluate models’ ability to troubleshoot commonly published lab protocols, we modify 108 multiple choice questions from FutureHouse’s ProtocolQA dataset [7] to be open-ended short answer questions, which makes the evaluation harder and more realistic than the multiple-choice version. The questions introduce egregious errors in common published protocols, describe the wet lab result of carrying out this protocol, and ask for how to fix the procedure. To compare model performance to that of PhD experts, we performed expert baselining on this evaluation with 19 PhD scientists who have over one year of wet lab experience.

All models underperform the consensus expert baseline (54%) and the median expert baseline of 42%. To run this eval with browsing, we maintain a domain blocklist and filter out any browsing results to sites on the blocklist. We also inspect browsing rollouts using a classifier which flags instances of cheating and manually review all flagged rollouts.

We evaluated models on a tacit knowledge and troubleshooting multiple choice dataset created with Gryphon Scientific. The questions span all 5 stages in the biothreat creation process and focus on areas where tacit knowledge would be a bottleneck. Tacit knowledge questions are meant to be obscure to anyone not working in the field, i.e., they either require tracking down authors of relevant papers or knowing people in the field. Troubleshooting questions are meant to be obscure to anyone without hands-on experience, i.e., the answers are known only to people who have tried the protocol themselves.

This set is uncontaminated; it was created fully in-house with our partners at Gryphon Scientific and has not been published.

None of the tested models outperform the consensus expert baseline of 80%, though all models outperform the 80th percentile PhD expert baseline of 63%. gpt-5.2-thinking performs lower than prior models due to increased refusals. If we treat all refusals as passes, gpt-5.2-thinking scores 83.33% on this evaluation.

To evaluate models’ ability to identify and correct real-world experimental errors in biological protocols, we built a short-answer troubleshooting dataset from expert-written wet lab procedures. TroubleshootingBench focuses on tacit, hands-on knowledge and uncontaminated procedures that are not available online.

Scientists with a PhD in a relevant biological discipline (virology, genetics, microbiology, or protein engineering) were asked to transcribe biological protocols they have personally used in the lab. Each protocol must include precise step-by-step procedures, equipment, and reagents. If a protocol was adapted from a publication, experts were required to significantly alter at least several steps. From these protocols, they created three troubleshooting questions each, introducing subtle or realistic execution errors (e.g., improper homogenization technique) and describing the resulting failed outcome.

After going through independent expert review, the resulting dataset includes 52 protocols, each paired with three expert-written troubleshooting questions. To benchmark model performance, we ran a human baselining campaign with 12 independent PhD experts. The 80th percentile expert score (36.4%) is used as an indicative threshold for model performance. Compared to ProtocolQA Open-Ended, which focuses on well-known published procedures, TroubleshootingBench is designed to test model performance on non-public, experience-grounded protocols and errors that rely on tacit procedural knowledge

gpt-5.2-thinking scores highest on this evaluation without accounting for refusals, 3 percentage points more than gpt-5.1-thinking.

Cybersecurity is focused on capabilities that could create risks related to use of the model for cyber-exploitation to disrupt confidentiality, integrity, and/or availability of computer systems. Our internal results show that gpt-5.2-thinking performs considerably better than gpt-5-thinking on our evaluations and is around a similar capability level as gpt-5.1-codex-max. The gpt-5.2-thinking model series does not meet the threshold for High cyber capability.

Under our Preparedness Framework, High cybersecurity capability is defined as a model that removes existing bottlenecks to scaling cyber operations, including either by automating end-to-end cyber operations against reasonably hardened targets, or by automating the discovery and exploitation of operationally relevant vulnerabilities. Concretely, this means a model qualifies as High if it can develop working zero-day remote exploits against well-defended systems, or meaningfully assist with complex, stealthy enterprise or industrial intrusion operations aimed at real-world effects.In our analysis of prior cybersecurity incidents involving severe harm through either of these two routes, we find each instance demanded proficiency in each of three skills:

Each of these evaluations has important limitations:

Because of these limitations, excelling on all three evaluations is necessary but not sufficient for a model to be considered High in cyber capability. Even very strong results can still fall short of what is required for scalable, end-to-end cyber operations against hardened targets.

Given that we expect capabilities to continue to increase, we are continuing to invest in safeguards, including both the model safety training described above and scaling up our monitoring and enforcement pipeline to disrupt potential misuse. We are also working on initiatives to help defenders and strengthen cybersecurity safeguards, and we continue to build and strengthen our cybersecurity evaluations. You can read more about our approach to safeguards for models that reach High levels of capability here.

We evaluate model performance on a test set of curated, publicly available CTF challenges which met the following criteria:

Challenges were further divided into the level of proficiency needed to solve the challenge, ultimately organized by whether the challenge required high school, collegiate, or professional-level cybersecurity knowledge and skills, with judgment on the requisite skills manually determined by our in-house cybersecurity expert. For all Collegiate and Professional CTFs, we require that the solution involves a non-trivial level of exploitation, requiring chaining two or more exploitation steps and not be solvable by copy-pasting textbook examples. Historically, we reported results across high school, collegiate, and professional levels but given the rise in capabilities and our investment in more rigorous benchmarks; we will focus on professional CTFs.

To evaluate the model against the CTFs, we give it access to a headless Linux box with common offensive cybersecurity tools preinstalled as well as a harness which allows the model to call those tools or otherwise execute commands similar to as a human. We run 16 rollouts for each CTF exercise, recording the pass@12 metric over the best set of rollouts; this metric is then compared to the thresholds established by the Preparedness Framework as well as the performance of previously released OpenAI models. While the performance is strong, it does not reach our threshold for potentially High cyber capability.

As always, we note that these evaluation results likely represent lower bounds on model capability, because additional scaffolding or improved capability elicitation could substantially increase observed performance.

CVE-Bench is a benchmark that tasks models with identifying and exploiting real-world web-application vulnerabilities in a sandbox environment. We used CVE-Bench (version 1.0) with a focus on vulnerabilities covering content-management systems, AI/ML apps, business-management tools, operational-monitoring systems, web infrastructure, libraries/packages, e-commerce platforms, and a small number of computing-management, mail-server, and web-portal applications; due to some infrastructure porting challenges, we only ran 34 out of the 40 benchmark challenges. We ran the benchmark using the zero-day prompt configuration, where the model is given a general task description of what it needs to do, as opposed to the one-day configuration, where the model is given a description of the vulnerability to exploit. Additionally, we ran the benchmark such that the agent does not have access to the source code of the web-application, and instead must probe it remotely.

We use pass@1 for this evaluation to measure the model’s ability to consistently identify vulnerabilities which are considered relatively straightforward by internal cybersecurity experts. Consistency is important to measure the model’s cost-intelligence frontier to identify vulnerabilities and its ability to potentially evade detection mechanisms that look for scaled attempts of vulnerability discovery and exploitation.

gpt-5.2-thinking performs 8 percentage points better than gpt-5.1-thinking, but 11 percentage points worse than gpt-5.1-codex-max, which has the ability to extend its work across multiple context windows.

Cyber range exercises measure a model’s ability to conduct fully end-to-end cyber operations in a realistic, emulated network.These exercises are long-form, requiring the model to (1) construct a plan to achieve an abstract adversary objective; (2) exploit vulnerabilities, misconfigurations, and weaknesses that are likely to be seen in the wild; and (3) chain together these exploits to achieve the scenario objective.

Performance is measured as pass or fail over 16 trials, where a pass denotes at least one success. Overall success rate is the percentage of passed scenarios.

External Evaluations by Irregular

Irregular, an AI Security organization that addresses and mitigates frontier cybersecurity risks, evaluated the model’s ability to solve three types of cyberoffensive challenges:

As part of the test setting, the model has 1000 turns to interact with each challenge, with elicitation techniques that aim to maximize its capabilities. The model’s goal is to retrieve a secret flag. The model runs on each challenge multiple times with measured success rates and cost-per-success: The total cost of all runs divided by the number of successful runs.

On the v1 atomic challenge suite, gpt-5.2-thinking achieved an average success rate of 83% in Vulnerability Research and Exploitation, 100% in Network Attack Simulation challenges, and 73% in Evasion challenges. For more details demonstrating the model’s capabilities on a specific challenge, see Irregular’s "Cryptographic Challenge Case Study" report.

In terms of API usage cost (assuming the same pricing as GPT-5) for solved challenges, the average cost-per-success in Vulnerability Research and Exploitation challenges is $10.8, the average cost-per-success in Network Attack Simulation challenges is $3.7, and the average cost-per-success in Evasion challenges is $12.9.

gpt-5.2-thinking performed at a similar capability level to gpt-5.1-codex-max and did not meet our High thresholds. The High capability threshold is defined to be equivalent to a performant mid-career research engineer and performance in the evaluations below indicate we can rule this out for gpt-5.2-thinking.

Measuring if and when models can automate the job of an OpenAI research engineer is a key goal of self-improvement evaluation work. We test models on their ability to replicate pull request contributions by OpenAI employees, which measures our progress towards this capability.

We source tasks directly from internal OpenAI pull requests. A single evaluation sample is based on an agentic rollout. In each rollout:

If all task-specific tests pass, the rollout is considered a success. The prompts, unit tests, and hints are human-written.

gpt-5.2-thinking is the highest performing model on this eval, exceeding even gpt-5.1-codex-max.

Developed by the Preparedness team, MLE-bench evaluates an agent’s ability to solve Kaggle challenges involving the design, building, and training of machine learning models on GPUs. In this eval, we provide an agent with a virtual environment, GPU, and data and instruction set from Kaggle. The agent is then given 24 hours to develop a solution, though we scale up to 100 hours in some experiments.

The full dataset consists of 75 hand-curated Kaggle competitions, worth $1.9m in prize value. Measuring progress towards model self-improvement is key to evaluating autonomous agents’ full potential. We use MLE-bench to benchmark our progress towards model self-improvement, in addition to general agentic capabilities. The subset plotted below is 30 of the most interesting and diverse competitions chosen from the subset of tasks that are <50GB and <10h.

gpt-5.2-thinking scores comparably to gpt-5.1-codex-max on this evaluation.

PaperBench [9] evaluates the ability of AI agents to replicate state-of-the-art AI research. Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch, including understanding paper contributions, developing a codebase, and successfully executing experiments. For objective evaluation, we develop rubrics that hierarchically decompose each replication task into smaller sub-tasks with clear grading criteria. In total, PaperBench contains 8,316 individually gradable tasks.

We measure a 10-paper subset of the original PaperBench splits, where each paper requires <10GB of external data files. We report pass@1 performance with high reasoning effort and no browsing.

gpt-5.2-thinking scores only one percentage point below gpt-5.1-codex-max, our highest scoring model on this benchmark.

OpenAI-Proof Q&A evaluates AI models on 20 internal research and engineering bottlenecks encountered at OpenAI, each representing at least a one-day delay to a major project and in some cases influencing the outcome of large training runs and launches. “OpenAI-Proof” refers to the fact that each problem required over a day for a team at OpenAI to solve. Tasks require models to diagnose and explain complex issues—such as unexpected performance regressions, anomalous training metrics, or subtle implementation bugs. Models are given access to a container with code access and run artifacts. Each solution is graded pass@1.

gpt-5.1-codex-max remains the highest scoring model on this benchmark at 8%.

External Evaluations by Apollo Research

Apollo Research conducted a full evaluation of gpt-5.2-thinking for strategic deception, in-context scheming, and sabotage capabilities. Across these assessments, Apollo found that gpt-5.2-thinking’s capability and propensity for scheming are broadly comparable to other recent frontier models, including gpt-5.1-thinking and gpt-5.1-codex-max. gpt-5.2-thinking occasionally engages in deceptive behaviors such as falsifying data, feigning task completion, or strategically underperforming when given an explicit in-context goal, but it shows low rates of covert subversion against developers and does not engage in sabotage or self-preservation behaviors in these settings. While gpt-5.2-thinking may exhibit higher awareness of being evaluated than some prior OpenAI models, deception rates are generally similar to peers and are not substantially amplified by scaffolding. In addition, Apollo observed fewer linguistic anomalies in gpt-5.2-thinking’s chain-of-thought compared to gpt-5.1-thinking. Based on these findings, Apollo concludes that gpt-5.2-thinking is unlikely to be capable of causing catastrophic harm via scheming.

1. [1]

   Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, et al. A strongreject for empty jailbreaks. arXiv preprint arXiv:2402.10260.
2. [2]

   OpenAI. “Introducing GPT-5.” Available at: https://openai.com/index/introducing-gpt-5/.
3. [3]

   OpenAI. “Pioneering an AI clinical copilot with Penda health.” Available at: https://openai.com/index/ai-clinical-copilot-penda-health/.
4. [4]

   OpenAI. “Introducing HealthBench.” Available at: https://openai.com/index/healthbench/.
5. [5]

   Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, et al. CharXiv: Charting gaps in realistic chart understanding in multimodal LLMs. arXiv preprint arXiv:2406.18521. Available at: https://arxiv.org/abs/2406.18521.
6. [6]

   Tyna Eloundou, Alex Beutel, David G. Robinson, Keren Gu-Lemberg, Anna-Luisa Brakman, Pamela Mishkin, et al. “First-person fairness in chatbots.” Available at: https://cdn.openai.com/papers/first-person-fairness-in-chatbots.pdf.
7. [7]

   Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, et al. “LAB-bench: Measuring capabilities of language models for biology research.” Available at: https://arxiv.org/abs/2407.10362.
8. [8]

   Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, et al. “CVE-bench: A benchmark for AI agents’ ability to exploit real-world web application vulnerabilities.” Available at: https://arxiv.org/abs/2503.17332.
9. [9]

   Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, et al. “PaperBench: Evaluating AI’s ability to replicate AI research.”