One Benchmark Replayed $57.4 Million of Exploits. Another Found No Complete Response Plan.

An AI agent reproduced $57.4 million of simulated historical exploit value across 200 smart-contract cases at a reported average API cost of $2.39 a case. In a separate test of compromised cloud hosts, none of 23 tested models delivered complete detection and remediation planning on any one of 10 ranges (CyberChainBench; SecRespond).

The figures point in the same direction: narrow, replayable attack work is ahead of complete defensive response. They do not prove an autonomous attacker would beat a security team. The two preprints use different models, scaffolds, inputs, environments and scoring rules. One runs code against historical blockchain forks. The other grades forensic reports and proposed fixes on frozen host snapshots.

That distinction is the story. Exploitation can be reduced to a bounded objective with rapid machine feedback. Incident response must find what is missing, preserve normal operations and know when the job is finished.

Exploitation gets a tight target

CyberChainBench starts with 541 disclosed smart-contract incidents across nine EVM-compatible chains. Every case is anchored to historical chain state. The detection task covers all 541 cases. The exploit experiment uses 200 cases. Patching is limited to 94 cases with upgradeable proxies, verified source and suitable historical transactions for regression tests (preprint; repository).

Its top Codex and GPT-5.5 configuration scored 37.5% on detection, 43.7% on exploitation and 23.4% on patching. Those percentages have different denominators and meanings. Detection requires an exact match on vulnerability type and function. Exploitation is the average ratio of agent-generated profit to the historical reference profit, capped at one. Patching passes only when the old exploit is blocked and all selected legitimate transactions still work.

The $57.4 million is simulated historical exploit value, not stolen money, current profit or a forecast. The agent ran in isolated containers against old blockchain state. Its exploit task also received the correct vulnerability type and function, then could call a validator repeatedly within a 30-minute budget. It did not have to discover the target from scratch.

This setup makes the result useful for measuring exploit construction. It makes it weak evidence about end-to-end autonomous attacks. The benchmark authors also warn that models may have encountered public incident reports or DeFiHackLabs reproductions during training. Detection scores fell on 86 cases dated after model knowledge cutoffs, although contamination could not be ruled out.

Defense has an open-world stopping problem

SecRespond tests a different failure mode. Its 10 ranges use frozen snapshots of deliberately compromised cloud hosts covering four entry-point types, 21 MITRE ATT&CK techniques and five operating systems. All 23 models run through OpenCode with the same task inputs and tools (preprint; released benchmark).

Agents receive alerts, vulnerability scans and baseline checks. The range designers arranged for only 30% to 40% of attack actions to trigger alerts. The remaining 60% to 70% leave silent file or configuration evidence. Models must inspect the disk, reconstruct the attack and produce four reports plus a remediation plan.

The benchmark does not ask agents to execute those repairs on a live host. Three model judges grade 280 expert-written checkpoints for detection and planning. A human expert separately graded 60 checkpoints across 10 trajectories. The reported correlation with the model-judge average was 0.96, but that validation supports the rubric, not real-world recovery.

No model reached complete detection and remediation planning on a single range. The strongest overall entry, Claude Opus 4.7, averaged 79.0% for detection and 65.7% for planning across the nine ranges it completed. It refused the NPM-Worm range. GPT-5.5 averaged 70.7% and 36.0%, respectively. Models followed exposed alerts, missed quiet persistence and proposed the first plausible cleanup without consistently rotating credentials, blocking outbound channels or checking service health.

The shared GPT-5.5 label does not create a controlled comparison. CyberChainBench paired it with Codex. SecRespond paired it with OpenCode. The first exploit task supplies localization hints and returns executable profit feedback. The second asks for open-ended forensic coverage, then uses a rubric to grade written output. A common model cannot cancel those differences.

A firmer head-to-head test would use one agent scaffold, equal time and tool budgets, and overlapping cases that require discovery, exploit construction, executed repair and regression checks. It would measure both harm reproduced and harm prevented. Neither paper does that. SecRespond's human validation strengthens confidence in its checkpoint grades, but it cannot turn a planning score into evidence of a repaired host.

Exploit replay asks whether one path works. Incident response asks whether every relevant path has been found, closed and checked for collateral damage.

The asymmetry is operational, not a race result

The strongest counter-reading is also the most useful. Narrow exploit replay is easier to score than forensic investigation and regression-free recovery. CyberChainBench supplies localization hints to the exploit task and an executable profit oracle. SecRespond withholds silent evidence and asks agents to establish completeness from an open filesystem. The benchmark design creates part of the gap because real attack and defense work have different stopping conditions.

Yet each study finds a similar imbalance inside its own framework. CyberChainBench's best patch score trails its exploit score even though the patch task receives attack traces. SecRespond's planning scores trail detection for every model. Adding an incident-response skill improved several SecRespond results, especially planning, but also caused regressions on broad ranges when models stopped at the skill's boundaries.

Security buyers should therefore ask for evidence around the model, not a model rank. An exploit agent needs isolation, target authorization and controls against live endpoints. A response agent needs exhaustive host enumeration, executed remediation, regression tests, credential rotation checks and a human-verifiable evidence trail. Neither preprint demonstrates that full production loop.

The present advantage is not that agents are inherently better at attacking. It is that a narrow objective can be validated quickly, while complete repair still depends on proving a negative: nothing important was missed.