CyberArena Tests Whether Security Agents Can Attack and Defend at Once

Security benchmarks often ask an AI agent to retrieve a flag from a fixed target. That setup is convenient and repeatable, but it measures only one side of real security work: finding a path through a system that never responds. CyberArena proposes a more adversarial test. Two agents receive identical vulnerable services, then attack each other while patching their own copies. The result is less like an exam and more like an attack-and-defense capture-the-flag match.

The benchmark's central question is not simply whether an agent can discover an exploit. It is whether the agent can manage a changing system under pressure: identify weaknesses, close them without breaking legitimate behavior, notice what an opponent is doing, and adapt over multiple rounds. That combination makes the evaluation relevant to a broader idea of agent competence—acting safely and effectively in an environment where another capable actor changes the problem.

A benchmark built around interaction

Each CyberArena match consists of three 20-minute rounds. Both entrants start with the same challenge service and have root access to their own copy. At the same time, each can reach the opponent's service over a separate attack network. Fresh objectives are planted for every round, so a previously captured flag cannot simply be submitted again. Defensive patches persist, however, making the state of the service—and the available attack surface—different as the contest proceeds.

The arena, rather than either agent, determines the outcome. It validates submitted objectives and independently checks service health. Those checks matter because a trivial defense such as blocking all traffic could stop an attacker while also making the application useless. CyberArena therefore rewards three dimensions: capturing objectives, preventing the opponent from capturing them, and preserving availability.

This design turns patch quality into a first-class capability. An edit must address the underlying route to compromise while leaving expected workflows intact. Restart speed also matters: a correct fix that is applied late leaves more time for the opponent to exploit the service.

What the reported campaign measures

The published campaign covers 466 valid matches involving 12 frontier agents and 40 challenges. Because running every possible pairing on every task would be expensive, the benchmark observes a selected set of matches and uses a statistical model to estimate standings on a shared reference. Its inputs are externally observed round captures and health-check results, not self-reported success from the agents.

The source reports GPT-5.6 Sol at the top of this campaign with a score of 78.6, five points ahead of Claude Opus 5. The two leading systems retained their objectives in 83% and 81% of rounds respectively, while the remaining entrants ranged from 43% to 61%. These are results for this benchmark configuration and campaign, not a universal ranking of security ability. Harness choices, challenge composition, time limits, and the available tools all shape what is being measured.

The more useful lesson may be where separation appears. A static challenge can saturate when several models know or rediscover the same exploit. In a live match, an exploit may stop working after a patch, while a rushed defense may open another problem or fail a health check. The opponent effectively raises the ceiling of the task.

Failure modes become visible

CyberArena's audit highlights recognizable operational mistakes. The most frequent reported issue was claiming a fix without verifying it, seen in 23% of audited matches. Partial patches followed at 21%. The benchmark also describes over-blocking: eliminating attack paths by damaging the service itself. These failures are informative because they distinguish producing a plausible remediation from completing a reliable remediation loop.

For anyone evaluating an agent, that loop suggests practical questions beyond a final score. Does the system reproduce the vulnerability before editing? Does it identify all routes that share the same root cause? Does it run targeted checks after a restart? Can it use evidence from incoming attacks without becoming distracted? And does it preserve required behavior while reducing exposure? Match replays and per-round traces can make those questions easier to investigate than a single pass/fail flag.

How to read the benchmark responsibly

Live competition adds realism, but it does not recreate an enterprise security operation. CyberArena uses deliberately vulnerable services, bounded objectives, fixed rounds, controlled networking, and automated health checks. Real environments include incomplete inventories, ambiguous ownership, longer time horizons, social constraints, and consequences that cannot be compressed into a game score. The benchmark should therefore support controlled comparative evaluation, not justify unattended deployment against production systems.

It is also worth separating offensive success from defensive reliability. A system that captures many objectives may still make risky patches; a conservative defender may preserve availability but miss vulnerabilities. The combined score is useful for standings, while the component outcomes and traces are more useful for selecting an agent for a particular workflow.

CyberArena's strongest contribution is this shift in evaluation unit: from solving an isolated puzzle to sustaining useful behavior against an adaptive opponent. For researchers, it offers a way to test whether improvements survive contact with changing conditions. For practitioners, it provides a structured reminder that a security agent should be judged not only by what it finds, but by what it changes, what it verifies, and what still works afterward.