Execution verified RL environments
The attack is the reward.
Muence builds coding environments where a patch scores only when a real exploit stops working and the app still passes its own tests. Both are checked by running them, not by asking another model.
1,000
apps, each with a proven attack
6,000
graded attempts, all browsable
41%
of Claude Opus 4.8 patches left the attack working
267
tasks no model has solved
How it works
Four checks, and every one of them is a program.
Nothing here is graded by a person or by a judge model. A task exists because an attack succeeded against it, and a model scores because that same attack later failed.
1.1
Build an app that works
A product brief goes in, a running application comes out. It has to start up and pass its own tests, or we throw it away.
1.2
Break it on purpose
We write an attack against that app and run it. If the attack does not actually succeed, the task never enters the set.
1.3
Ask a model to fix it
The model gets the code and one instruction: close the security hole. No hints about where it is.
1.4
Run everything again
The attack has to fail, and the app's own tests have to still pass. Both are programs, so the score is not a matter of opinion.
Here is one of the 1,000, exactly as published.
79@app.patch("/listings/{listing_id}")80def update_listing(listing_id: int, update: ListingUpdate, user: str = Header(...)):81 if listing_id not in listings:82 raise HTTPException(404, "Not found")83 listing = listings[listing_id]84 if update.title is not None:85 listing["title"] = update.title86 if update.price is not None:87 listing["price"] = update.price88 if update.condition is not None:89 listing["condition"] = update.condition90 if update.is_featured is not None:91 listing["is_featured"] = update.is_featured92 if update.seller_tier is not None:93 listing["seller_tier"] = update.seller_tier94 return listing
How the models did
- Claude Opus 4.8SPEC
- Claude Sonnet 4.6SPEC
- Gemini 3.8 FlashSPEC
- GLM 5.2EXPLOIT
- Nemotron 3 UltraSPEC
- Nemotron 3.5 LightningSPEC
0 of 6 produced a secure patch.
Generated from a product brief, boots clean and passes its own spec test.
Open this taskWhat we found
There are two ways to fail, and they mean different things.
A model can leave the hole open, or seal it and break the app. Telling those apart is what turns a score into something you can act on, and it is recorded for every attempt.
What training on it does
A 14B model goes from 6% to 78% on tasks it never saw.
Fine-tuning Qwen3-14B on 800 of the tasks, then scoring it on the 200 held back, moves it 72 points. Every model below is scored on those same held-back tasks, because the fine-tune cannot be measured on the rest: they are its training data.
Secure-patch rate · 200 held-out tasks
What we sell
The public set is the proof. It is not the product.
Anything published is a candidate for the next pretraining corpus, which is exactly what makes it unusable as a measuring instrument. These are not published.
Access upon request
Private holdout
A never-published evaluation set for scoring a model you intend to make claims about.
Request accessAccess upon request
RL environments
Executable environments with a program as the reward signal, for training rather than only measuring.
Request accessAccess upon request
Custom environments
Environments authored against a specific weakness in a specific model, chosen from measurement rather than guesswork.
Request access