ERPBench Changes Its AI Winner When Rivals Change. Its Release Cannot Reproduce the Test

ERPBench changes the leading AI model when it changes the competition. DeepSeek leads the preprint's fixed-opponent setting with a mean terminal valuation of 252.29 million and a mean rank of 1.67. Gemini leads the shared-model setting at 263.95 million and 1.76. The authors report the same task-level winner on only 21 of 100 matched problems.

That result would make competitive context part of an enterprise benchmark score. It would not establish that Gemini is the better enterprise model. More importantly, the code and data release linked by the paper does not yet contain the matched experiment needed to reproduce the claim.

The measurement idea is strong. The published evidence is not ready for procurement.

Same task, different market

The ERPBench preprint puts an AI agent in charge of a simulated cereal manufacturer. It must set prices, production, procurement, marketing and finance decisions for 12 products across three sales channels. Decisions run for six rounds of 30 simulated days. Inventory, capacity, cash and competitors all affect the terminal company valuation.

The experiment uses six model families: Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro and Qwen 3.7 Max.

Each model faces the same 100 problem configurations twice. In Solo, one evaluated model competes with five deterministic rule-based opponents. In Arena, all six evaluated models share one market. That design produces 600 model trajectories in each setting. Six decision rounds across 1,200 trajectories produce 7,200 model-level rounds.

The authors' reported averages shift sharply. Gemini gains 94.54 million in Arena. DeepSeek loses 17.58 million and Qwen loses 18.74 million. The other three changes are smaller.

The 21 shared winners imply 79 different winners by subtraction. The paper also reports a mean within-problem Spearman rank correlation of 0.182. Gemini finishes last on 22% of Solo problems and none of the Arena problems. This is broad ranking instability inside the reported panel, not merely one large Gemini outlier.

A leaderboard can measure its opponents

Most enterprise evaluations treat the environment as test furniture. ERPBench makes it an experimental variable. That is the useful contribution.

A pricing agent does not act against a neutral background. Rival prices change demand. Demand changes inventory. Inventory changes production and cash. If competitors also adapt, the same action can create a different outcome even when the focal model and business problem stay fixed.

For buyers, the implication is narrower than the model rankings and more durable. A leaderboard built against scripted competitors may select an independent optimizer. A shared market may reward response to changing peers. Neither setting represents enterprise quality in general. Each answers a different conditional question.

The preprint cannot establish why Gemini improves. It tests one six-model Arena composition for six rounds. The paper itself says model endpoints and service-side routing may change, while family labels do not guarantee identical vendor snapshots. It reports no sensitivity results for repeated stochastic runs of each problem, longer horizons, different mixes of model rivals or alternative fixed-opponent sets.

Its confidence intervals answer a different question. The authors resample problems, preserving each Solo and Arena pair, to estimate how the average ecology effect varies across this task panel. That does not measure whether a fresh API call on the same problem produces the same action or winner. For stochastic model services, task uncertainty and run-to-run uncertainty are separate risks.

Those are not minor extensions. They separate an ecology effect from a particular cast of competitors, run length and service route.

The release does not match the paper

The linked repository's release branch ends at commit 667e7e5, dated 5 June 2026. The preprint was submitted on 4 September. The gap shows in the artifacts.

The release's 100-problem allocation describes 60 Solo problems plus 40 Arena problems with zero overlap. That is the opposite of the paper's two matched 100-problem panels. The available Solo results contain 399 rows across 75 problem identifiers, with unequal counts by model and duplicate model-problem pairs. The Arena file contains 216 rows across 36 problems. Only 15 available problem identifiers intersect.

The code diverges too. The release's benchmark command configures Solo with one team, not one evaluated model plus five bots. Its bot strategy module implements only a conservative policy, although the paper names conservative, aggressive, balanced, growth-oriented and risk-averse opponents.

The reported aggregate table is internally consistent. The released records cannot independently verify it. The 79-task headline must therefore remain an attributed preprint finding, not a reproduced result.

What a buyer should demand

An evaluation lead can use ERPBench's design without adopting its winner. Run the same operational cases against at least two competitor populations. Repeat each case across seeds. Extend the horizon. Rotate rival models and scripted policies. Report rank stability alongside mean score.

The run manifest matters as much as the leaderboard. It should pin code, problem IDs, model endpoint versions, service routes, decoding settings and raw round-level traces. A benchmark that measures interaction needs enough provenance to distinguish model behavior from opponent composition and API drift.

ERPBench identifies a real blind spot: enterprise-agent rankings can be conditional on who else is in the room. Until its matched panel is released, it also demonstrates the corresponding audit rule. When the environment becomes part of the score, the environment must become part of the evidence.