GPT-Rosalind-6 System Card Addendum

To evaluate models’ ability to troubleshoot commonly published lab protocols, we modify 108 multiple-choice questions from FutureHouse’s ProtocolQA dataset [3] to be open-ended short answer questions, which makes the evaluation harder and more realistic than the multiple-choice version. The questions introduce egregious errors in common published protocols, describe the wet lab result of carrying out this protocol, and ask how to fix the procedure.

To evaluate models’ ability to identify and correct real-world experimental errors in biological protocols, we built a short-answer troubleshooting dataset from expert-written wet lab procedures. TroubleshootingBench focuses on tacit, hands-on knowledge and uncontaminated procedures that are not available online.

Scientists with a PhD in a relevant biological discipline were asked to transcribe biological protocols they have personally used in the lab. From these protocols, they created troubleshooting questions, introducing subtle or realistic execution errors and describing the resulting failed outcome. Compared to ProtocolQA Open-Ended, which focuses on well-known published procedures, TroubleshootingBench tests performance on non-public protocols and errors that rely on tacit procedural knowledge.

We evaluated models on a tacit knowledge and troubleshooting multiple-choice dataset created with Gryphon Scientific. The questions span all five stages in the biothreat creation process and focus on areas where tacit knowledge would be a bottleneck. Tacit knowledge questions are meant to be obscure to anyone not working in the field. Troubleshooting questions are meant to be obscure to anyone without hands-on experience.

The refusal-adjusted GPT-Rosalind-5.5 score published in its system card used a different calculation and is therefore omitted from this comparison.

To evaluate models’ ability to troubleshoot wet lab experiments in a multimodal setting, we evaluate models on a set of fully held-out virology troubleshooting questions from SecureBio, in a multi-select format. A response receives credit only if it selects every correct statement and no incorrect ones.

After adjusting for refusals, GPT-Rosalind-6 exceeds the indicative High capability thresholds on three of the four evaluations. We therefore continue to treat the model as High capability. Compared with the reported GPT-Rosalind-5.5 results, observed scores are higher on three evaluations and lower on Biorisk knowledge.

Predicting whether an AAV capsid variant will package successfully could reduce trial and error in viral-vector engineering, which we treat as a proxy for some dual-use biological design capabilities. We evaluate this capability using an evaluation developed by Dyno Therapeutics. The model predicts packaging scores for held-out sequences using labeled examples, protein embeddings and computational tools. Performance is measured using Spearman rank correlation.

Predicting how changes to protein sequence affect function could reduce trial and error in protein engineering, which we treat as a proxy for some dual-use biological design capabilities. We evaluate this capability using SHP2, a well-studied human protein. The model uses available experimental measurements to predict held-out functional assay results for single-amino-acid variants. Performance is measured by mean R² across three unpublished assay datasets.

Reliable predictions of cell entry could help researchers prioritize laboratory experiments in studies of virus-host interactions, but could also be a threat-relevant capability. We ask models to predict and rank held-out measurements from an existing, unpublished academic dataset. The underlying experiment studied existing spike proteins using replication-incompetent pseudoviruses. It did not introduce new mutations or generate infectious progeny.

We assess the ability to predict biological evolution using data from an existing but unpublished experimental phage-plasmid co-evolution system. Given observations available at each experimental checkpoint, the model predicts which mutations will subsequently appear. Performance is measured by the negative log-likelihood of observed mutations, with lower scores indicating better predictions. The experiment uses viruses that infect bacteria rather than humans. We treat performance as a proxy for some dual-use biological design capabilities.

None of the four Critical capability evaluations met its indicative threshold. Results relative to GPT-Rosalind-5.5 were mixed. We also rely on the parent model’s helpful-only evaluations and external biological capability assessments, as described in the GPT-6 Astra system card.

GPT-Rosalind-6 is designed to be more useful for legitimate advanced biological research than generally available ChatGPT models. This supports advanced life sciences workflows, including some dual-use topics, that GPT-6 Astra may refuse in General Access. We still tune the model to refuse or safely respond to requests that would continue to meaningfully enable severe biological weaponization. This training is done separately from the training of our classifiers and offline mitigations to decorrelate our safeguards.

To evaluate the quality of these model-level refusals, we track the safety of model responses to prompts that originate from held-out synthetic data, red-teaming, and production data. These benchmarks and their grading continue to be updated, meaning results do not directly translate to past system cards.

Table 10 reports results across our suite of biosecurity benchmarks, accounting for both model behavior and safeguards. We note that because GPT-Rosalind-6 is a trusted-access deployment, we mainly considered the results from the Biosecurity - Severe benchmark. Biosecurity - Benign evaluates model compliance (being fully helpful without refusals or caveats) on a set of benign biological and operational questions. The remaining benchmark scores show the fraction of model responses that pass the configured safety grader on prompts related to biological weaponization. A response that does not pass was flagged under this grading criterion; this does not necessarily mean it is unequivocally unsafe.

GPT-Rosalind-6 inherits the model and system level cyber safeguards of the GPT-6 Astra base model and additionally applies a more conservative cyber refusal boundary. The model is intended to continue to refuse requests that clearly enable unauthorized, destructive, or harmful cyber activity, including areas such as malware deployment, credential theft, exfiltration, or other malicious activity.

As an additional robustness test, we evaluate resistance to previously identified cyber jailbreaks by replaying nine known attacks across ten fixed ExploitBench tasks. We evaluated both models with real-time cyber blocking disabled, so the task success rates in Table 11 reflect model-level behavior only. We report the mean task success rate across seven attacks, excluding attempts that ended in errors or exceeded turn limits.

Two attacks were excluded because all attempts ended in errors or exceeded turn limits.

We also evaluate GPT-Rosalind-6’s instruction following, honesty, resistance to prompt injection, and behavior when using tools. Additional evaluations assess whether automated monitors detect harmful actions and whether the model can evade detection or alter its reasoning when instructed to do so.

In the tables below, ↑ indicates higher scores are better and ↓ indicates lower scores are better. GPT-6 Astra results are included for context. Evaluation versions, inference settings, grading and sample coverage differ across some runs, so these are descriptive comparisons. “Not available” indicates that no numerical reference result has been identified. ≈ denotes an approximate value read from a published figure.