Execution verified RL environments

The attack is the reward.

Muence builds coding environments where a patch scores only when a real exploit stops working and the app still passes its own tests. Both are checked by running them, not by asking another model.

1,000

apps, each with a proven attack

6,000

graded attempts, all browsable

41%

of Claude Opus 4.8 patches left the attack working

267

tasks no model has solved

01

How it works

Four checks, and every one of them is a program.

Nothing here is graded by a person or by a judge model. A task exists because an attack succeeded against it, and a model scores because that same attack later failed.

1.1

Build an app that works

A product brief goes in, a running application comes out. It has to start up and pass its own tests, or we throw it away.

1.2

Break it on purpose

We write an attack against that app and run it. If the attack does not actually succeed, the task never enters the set.

1.3

Ask a model to fix it

The model gets the code and one instruction: close the security hole. No hints about where it is.

1.4

Run everything again

The attack has to fail, and the app's own tests have to still pass. Both are programs, so the score is not a matter of opinion.

Here is one of the 1,000, exactly as published.

marketplace-listing-api-256cc5cfIDOR
Marketplace listing API
79@app.patch("/listings/{listing_id}")
80def update_listing(listing_id: int, update: ListingUpdate, user: str = Header(...)):
81 if listing_id not in listings:
82 raise HTTPException(404, "Not found")
83 listing = listings[listing_id]
84 if update.title is not None:
85 listing["title"] = update.title
86 if update.price is not None:
87 listing["price"] = update.price
88 if update.condition is not None:
89 listing["condition"] = update.condition
90 if update.is_featured is not None:
91 listing["is_featured"] = update.is_featured
92 if update.seller_tier is not None:
93 listing["seller_tier"] = update.seller_tier
94 return listing

How the models did

  • Claude Opus 4.8SPEC
  • Claude Sonnet 4.6SPEC
  • Gemini 3.8 FlashSPEC
  • GLM 5.2EXPLOIT
  • Nemotron 3 UltraSPEC
  • Nemotron 3.5 LightningSPEC

0 of 6 produced a secure patch.

Generated from a product brief, boots clean and passes its own spec test.

Open this task
02

What we found

There are two ways to fail, and they mean different things.

A model can leave the hole open, or seal it and break the app. Telling those apart is what turns a score into something you can act on, and it is recorded for every attempt.

Claude Opus 4.855.4% fixed properly
Gemini 3.8 Flash55.1% fixed properly
Claude Sonnet 4.632.8% fixed properly
GLM 5.227.9% fixed properly
Nemotron 3 Ultra27.4% fixed properly
Nemotron 3.5 Lightning25.6% fixed properly
fixed, app still worksattack still worksapp brokenother
See every attempt, task by task
03

What training on it does

A 14B model goes from 6% to 78% on tasks it never saw.

Fine-tuning Qwen3-14B on 800 of the tasks, then scoring it on the 200 held back, moves it 72 points. Every model below is scored on those same held-back tasks, because the fine-tune cannot be measured on the rest: they are its training data.

Secure-patch rate · 200 held-out tasks

Qwen3-14B (SFT on VibeSec)
78.0%
Claude Opus 4.8
57.5%
Claude Sonnet 4.6
33.5%
Kimi K2.7 Code
32.0%
GLM 5.2
30.0%
Nemotron 3 Ultra
26.0%
GPT-OSS 120B
9.0%
Mistral Medium 3.5
8.0%
Qwen3-14B (base)
6.0%
+72 pp over the base model on tasks it never trained on, and +20.5 pp over Claude Opus 4.8, the strongest frontier model on the same set. Claude Opus 4.8 scores 64.9% across all 1,000 public tasks; this split is harder than the full set for every model.
Read the method and the limitations