SemiAnalysis发布GDPx前沿模型基准测试

GDPx is a benchmark that covers the most valuable domains for frontier intelligence. It aims to be the world’s best benchmark for measuring general agentic capabilities.

We believe the first trillion dollars of AI revenue will be concentrated in sectors that can deploy frontier intelligence at high ROI, not broadly diffused throughout the whole economy. Building a general-purpose benchmark based on share of GDP or the SOC/O*NET occupational taxonomy is therefore a mistake. If your goal is to track the progress of frontier models, you should focus on the domains where they will be most impactful.

Specifically, we cover the following domains:

• Chip design
• Cybersecurity
• Biology
• Finance
• Legal
• Quant trading
• Recursive self-improvement (RSI)
• Hardware design
• Materials science
• Alignment
• Software engineering

And intentionally forgo industries like government, real estate, and insurance.

Interpreting the results

GDPx is a true frontier-level benchmark: even the best model scores just 12%.

Source: GDPx

It’s worth emphasizing that certain models like Opus 5.5 and GPT-6 Astra have sandbagged overall scores due to their safety classifiers. If we exclude cyber, biology, and alignment—the three domains where safety classifiers sometimes kick in—you get the following results instead.

Source: GDPx

We were pleasantly surprised by how well this ranking lines up with our subjective impression of the models. OpenAI and Anthropic generally dominate the top of the ranking while SpaceXAI, Meta, and open-weight models are all meaningfully behind. Gemini 4 Carbon is a real improvement from Google and the only model in the same ballpark as Opus and Astra. It will be interesting to see people’s impressions once Carbon becomes generally available.

We believe it’s underrated how relevant safety classifiers are when picking the best model for certain use cases. For example, Muse Wonder Full is actually the best available cyber model today by a meaningful margin.

Source: GDPx

We also generally observe meaningful spikiness across all leading models.

Source: GDPx

Opus 5.5 is the best all-around model and wins in quant trading, hardware design, and chip design. Gemini 4 Carbon is the best bio model (likely due to weaker safety classifiers) and edges out the win in software engineering. GPT-6 Astra wins convincingly for RSI and is also the best model for materials science. Our finance and legal tasks are so challenging that basically every model scores exactly 0.

Interestingly, if you subset to a single model family, you will sometimes see very similar spikes.

Source: GDPx

The cost vs performance Pareto frontier also matches our general experience using the models: Luna and DeepSeek fight for the low end, Opus wins the high end, and OpenAI claims the middle. Again, however, the appearance of Gemini 4 Carbon is worth watching closely.

Source: GDPx

It’s impossible to make all the nuanced comparisons enabled by GDPx in a single newsletter. To explore the data yourself, checkout the official website.

A collaboration with 14 specialized data partners

Domain-specific benchmarks have gotten significantly better over the past year largely thanks to the rise of specialized RL environment startups. Yet OpenAI’s GDPval (along with its derivatives like GDPval AA v2.1) remains the most popular general purpose benchmark still reported on major new model release cards. This is despite being released over a year ago on September 25, 2025—an eternity in AI land. GDPval may have been marketed as a “knowledge work” benchmark, but agents weren’t really even a thing yet. This had a huge impact on its task design, and as we’ll later explain, makes GDPval woefully inadequate for evaluating today’s frontier models.

Our goal with GDPx is to create the world’s best benchmark for general agentic capabilities. However, as soon as we started working on this project, it became clear why we’ve yet to see a worthy GDPval successor. Models like GPT-6 Astra and Opus 5.5 are already pushing the boundaries of human knowledge. If you want to make a finance or cyber task these models can only solve < 20% of the time, you need real domain expertise to ensure the task is hard for the right reasons. Many popular benchmarks sidestep this problem by shipping tasks that are downright unfair. This typically manifests in a grader that checks for requirements impossible to infer from the task description or environment. For example, a coding problem may ask the model to fix a bug that a human engineer would agree has two valid solutions—one using “module A” and another using “module B”. However, a poorly designed grader may require the AI to use “module A” and unfairly give 0 reward when the AI chooses “module B”. Correct all such issues, and the benchmark often ends up being saturated.

Making high quality tasks for a single domain you have expertise in is already hard enough—there’s a reason why labs are willing to pay $5k+ for a single high quality task. Finding one company with sufficient expertise in both AI evals and enough domains to make a broad-based GDP benchmark is downright impossible.

To solve this problem, SemiAnalysis partnered with specialized RL environment startups for each domain:

• Chip Design (Aria, Arcverex, Ergodex)
• Cybersecurity (Arithmetic)
• Biology (Latch Bio)
• Finance (Living Context)
• Legal (Chakra)
• Quant trading (Third Moment)
• RSI (Extrapolate)
• Hardware design (Beautiful Machines)
• Materials science (ValenceForge, Emulated)
• Alignment (Fleet)
• Software engineering (Datacurve)

All partners were subject to a rigorous QA process we detail in the methodology section. For some domains, we had multiple partners who wanted to be exclusive and ran a process to select the one with the highest quality tasks.

We would like to give a huge thank you to all of our partners for contributing to GDPx. They’re among the best vendors for their respective domains and it would’ve been impossible to create such a broad-based benchmark without their help.

Additionally, we’d like to thank Fireworks for providing inference for all the open source models, Modal for providing the sandbox infra needed to run our tasks, as well as Abhijay Rana for helping source some of the partners and assisting with early QA.

Problems with OpenAI’s GDPval

To be clear, we believe GDPval was impressive for its time and was the first benchmark to evaluate AI on such a broad range of tasks. It’s not OpenAI’s fault that a benchmark created before the rise of agents does a poor job evaluating them today.

Every GDPval task is precisely specified and non-interactive, with all the necessary context directly spoon-fed to the model in the prompt. Consider the following representative finance example below:

Source: SemiAnalysis

Attached to this prompt is a very plain, single-page PDF that gives the model even more context about the two projects.

An actual document used for an OpenAI GDPval finance task. Source: OpenAI

From here, the model is asked to directly one-shot its task of writing a 15 page pdf to recommend one of the two options.

Any real finance manager could point out dozens of reasons why this task isn’t actually representative of their day to day work, but we will focus on two main issues:

• Despite the task description itself being unnaturally wordy, the overall task lacks detail about the “two significant investment opportunities.” The model is expected to take nothing more than a ~500 word, high-level description of the two options and convert it into an “up to 15 page” pdf recommending one of them. As you’d expect, even OpenAI’s reference solution is mostly filler slop.