The FrontierPhysics benchmark found that no AI agent passed more than 46 of 168 attempts at end-to-end physics research. Across 22 models in 26 configurations, the best result came from Claude Opus 5.5 running in Claude Code, which passed 46 trials for a Pass@1 rate of 27.4 percent. The result matters for laboratories considering AI research tools because the test measured the full research loop, from literature review and planning through experiments, verification and manuscript preparation.
FrontierPhysics benchmark: A harder test of AI research
The benchmark was submitted to arXiv on November 4, 2026, by UC Berkeley doctoral candidate Bingran You and 71 co-authors. It contains 56 tasks built by 42 physicists from their own research, covering nine domains across theoretical, computational and experimental physics. Existing physics evaluations often focus on fixed questions, isolated sub-tasks or reproductions of published papers with solutions provided, according to the paper.
FrontierPhysics instead gives agents high-level research briefs and supporting materials without specifying the complete method or solution path. Each task requires an agent to review the literature, develop a research plan, conduct research and experiments, and deliver both structured numerical outputs and a scientific manuscript. The authors said the standard for publishability rests on task rubrics and an automated reviewer, not on journal peer review.
The tasks also reflect substantial human effort. Contributors submitted 182 candidate tasks, and 56 were accepted. Eighteen tasks build on work the contributors had already published, while four rest on Nature papers. On average, each task author has 420 citations, and contributors reported about 270 days of work per original project and more than 8,000 hours in total. Each underlying project was required to involve at least two weeks of research effort.
Correct answers are only part of the test
Grading has two parts. Deterministic verifiers check structured outputs against reference values, while a reviewer agent applies expert-authored rubrics to the manuscript and the trajectory of the work. Rubrics contain binary blocking criteria that operate as gates, so a rollout receives a zero score unless the verifier and every blocker pass. High quality on other criteria cannot compensate for a failed numerical check or blocker.
OpenAI’s GPT-5.6 Sol, run at its highest setting in the Codex command-line tool, reviewed every rollout, with a 10-hour timeout per attempt. The paper said this reviewer setup was checked against an expert audit and on the authors’ own papers. A later regrade using Claude Opus 5.5 produced nearly the same pass or fail decisions for GPT-6 Astra rollouts, but the Opus 5.5 reviewer was more lenient toward its own model family.
The benchmark used BenchFlow’s command-line tool to run experiments and track agent trajectories, verifier results and rubric results. Smaller jobs ran through the Daytona sandbox provider, while tasks requiring heavier resources used Docker in Azure virtual machines with Lambda API access. The paper does not clearly state whether all 56 tasks are public, though reviewer workspaces and trajectories were collected as Hugging Face datasets.
Uncertainty around the leader
Every configuration attempted all 56 tasks three times, producing 168 trials per configuration. The top configuration, Claude Opus 5.5 in Claude Code, passed 46 of 168 trials, a Pass@1 of 27.4 percent. Its approximate 95 percent binomial interval is plus or minus 6.7 percentage points. The wider task-clustered bootstrap interval runs from 17.9 percent to 36.9 percent, reflecting the relationship among the three related trials for each task.
The authors reported overall pass rates below 30 percent for every agent tested. That is a benchmark result, not proof that agents cannot perform useful physics work. It does show that solving a bounded question and executing a complete research project remain different challenges, with the latter exposing failures in validation, scientific judgment and communication.
OpenAI’s GPT-6 Astra at its highest setting initially passed 25 of 168 attempts, or 14.9 percent, before the authors corrected nine misgraded attempts. That correction lifted the manually adjusted tally to 34 of 168, or 20.2 percent. The reported final rate in the paper’s comparison table is 19.6 percent after reruns and regrading, a different stage of the same bookkeeping process.
Why agents failed
Task contributors and the admin team manually reviewed the 168 trials from GPT-6 Astra, hand-labeling 143 failed rollouts. Because a rollout could carry several labels, the percentages overlap rather than sum to 100. Across those failures, 48 percent showed non-rigorous reasoning such as circular self-verification and too few edge cases, while 34 percent showed missing physics intuition.
Other labels included literature problems at 27 percent, instruction-following problems at 26 percent, wrong physics at 16 percent and execution mistakes at 4 percent. The paper’s conclusion is that the main weakness is insufficiently rigorous reasoning and limited expert intuition, not gaps in physics knowledge or execution errors. The labels were hand-applied to GPT-6 Astra only, so other agents may fail in ways outside the taxonomy, while non-blocker scores remain subjective.
The paper illustrates missing intuition with Rydberg decay. In the example, agents do not consider Rydberg decay when analysing sources of error, something an atomic physicist would naturally weigh. The failure points toward judgment rather than recall. It suggests that text-trained systems may miss considerations embedded in expert practice, even when the underlying formulas remain accessible.
Different benchmarks measure different abilities
The score gap with CritPt illustrates why direct comparisons can mislead. CritPt, built by researchers at Argonne National Laboratory and the University of Illinois Urbana-Champaign, was released in September 2025. At release, the best base model reached 5.7 percent accuracy, and coding tools lifted results to about 10 to 13 percent. Artificial Analysis now lists GPT-6 Astra at 83 to 84 percent on CritPt v2, depending on the effort setting, while the same model passed 20.2 percent of FrontierPhysics attempts after correction.
Those figures do not mean GPT-6 Astra lost physics knowledge between evaluations. The two tests grade different work. CritPt checks answers to bounded research-level problems, while FrontierPhysics requires an unscripted process that must survive verifiers, blockers and expert rubrics. The scores sit on different scales and should not be treated as a like-for-like decline.
A Princeton-led study posted July 29 found a similar pattern in machine-learning research. Agents received the research questions from two unpublished NeurIPS 2026 papers, plus six days and thousands of dollars of compute. They completed the engineering, while the papers’ own authors rejected both outputs. That study also identified five recurring failure modes that a second model and scaffold reproduced, reinforcing concern about whether engineering completion translates into credible research.
What readers should take from the result
The practical implication is that researchers should tie claims of physics-research ability to the exact protocol being measured. A high result on a bounded benchmark does not establish competence in literature synthesis, experiment selection, error diagnosis or manuscript preparation. Human review may still add the most value where agents check their own work, cover edge cases and identify physical effects that are easy to overlook.
Bingran You is a doctoral candidate in applied science and technology at UC Berkeley working in Hartmut Haeffner’s lab, and a Member of Technical Staff at BenchFlow, which builds environments for AI agents. Gerbrand Ceder, a Berkeley materials scientist who co-founded Radical AI, is listed as a Kavli affiliate on the preprint. The FrontierPhysics benchmark therefore connects a wide expert network with infrastructure built around agent evaluation.
The overall message from the FrontierPhysics benchmark is narrow but important. Leading systems can complete many trials, yet fewer than three in ten pass the full protocol, and the leading result carries substantial uncertainty. The distance between a correct computation and publishable research remains the central gap, with rigorous self-checking and expert intuition emerging as the most important targets for future improvement.