Private holdout
A never-published evaluation set for scoring a model you intend to make claims about.
The public VibeSec set is on Hugging Face and in this site's data browser, which is exactly what makes it unsuitable for an official number: anything public is a candidate for the next pretraining corpus, and a benchmark you can train on measures memorisation rather than capability.
The holdout is generated by the same engine and graded by the same executable gates, and is never released. Tasks carry canary strings, so if the set ever does leak into a training corpus, that becomes detectable rather than invisible.
It is also deduplicated. The public set has seed fan-out, where one prompt produced many near-identical tasks, so a model that solves one variant tends to solve all of them and a raw pass rate over-weights whichever clusters happen to be large. Scoring on distinct problems is part of what you are paying for.
What you get
- A held-out task set, deduplicated by seed prompt and near-duplicate union
- Execution-verified grading: the exploit must stop firing and the spec test must still pass
- Failure attribution per task, separating an unfixed hole from a broken app
- Canary strings, so contamination is detectable later
- A scoring run on your model, with the per-class and per-failure-mode breakdown
Why this is not public
- Published sets can be trained on, and cannot be un-published
- The public set is a browsable proof of method, not a clean measuring instrument
The method is fully inspectable either way. Browse every public task, exploit and reference patch before you talk to us.