Every detection, sorted by how it was earned: blind and answer-hidden, or built to catch a bug that was already known. The distance between those two columns is the honest measure of the tool.
Compiled by Claude·from the project's own records·last updated 27 July 2026
A tool is worth exactly what it finds on something it has never seen. So this page keeps two kinds of result apart and never blurs them. The first is the blind test: real binaries, the known vulnerability hidden from the tool, no human steering. The second is the honest asterisk on every impressive demo, a detector built after the bug was known, aimed at the one binary it was meant to catch. Both are here. The gap is the story.
Blind and answer-hidden, MUGEN delivered 0 complete source-to-sink chains on the three CVE sites checked by hand. Every end-to-end catch it can show was engineered against a binary whose bug was already known.
full chain source to sink, deliveredsurfaced right function, no chainpartial site seen, not deliveredmiss not detectedfalse positive looked real, wasn'tsynthetic harness, not the field
By the numbers
Three pictures of the same split. The first two describe the blind backtest; the third is the whole page in one frame: what the tool finds when it has never seen the answer, against what it finds when the answer was known in advance.
Blind outcomes · 11 anchored targets
Not one produced a complete chain. Surfaced plus partial, 6 in all, means the right function was reached; the 5 misses found nothing at the site at all.
What became of the 41 acquired
Only 11 of the 40-one binaries could be anchored at all. 20-6 were stripped past the point the tool could even name the vulnerable function, which describes most real software in the wild.
The gap · complete source-to-sink chains delivered
The distance between these two bars is the honest measure of the tool. Blind, it delivered 0 complete source-to-sink chains. Every complete chain it can show was engineered to catch a bug that was already public.
The honest bar: a blind backtest
Real third-party binaries, each with a known CVE the tool was not told about. Static analysis only, 0 model tokens. 42 on the manifest, 40-one acquired; of those, 20-6 were stripped past the point the tool could even name the vulnerable function, 4 timed out, and 11 were anchored and scored. Here are all 11.
Not one delivered a complete source-to-sink path to the operator. On the three sites checked by hand, the count was 0. And these 11 are the easy slice: the 20-6 that were dropped were stripped of symbols, which describes most real software in the wild. The 0-token figure is literal: this run generated the operator's prompts without ever calling the model. The model, when it does run, only re-checks what the static pipeline already surfaced, so it could not have rescued a site the pipeline missed.
A kernel driver, calibrated
A single real target where the answer was known in advance, run with the debug symbols the tool almost never has in the field.
surfaced 12 of 15 patched functions reached the operator; function-level, not a delivered chain
36s
Surfaced means the patched function reached the top of the pile. It does not mean a source-to-sink chain was drawn. 12 surfaced, two fell past the ranking cut, and one missed on a name mismatch. And this ran with a version-matched symbol file that, by the project's own note, multiplies the detector's reach many times over what a stripped binary allows.
End-to-end, but not blind
End-to-end catches on genuine public exploit-database cases, delivered to the operator: at the #1 prompt in one case, lower in the queue or through a dedicated flag in the others. The caveat is the whole point of the section header. Each detector was built after the bug was known and aimed at that specific binary. This is "can the tool be made to catch a known bug end-to-end," not "point it at something new."
full chain the vulnerable callback promoted to the operator's #1 prompt; a second network sink in the same product was later confirmed by actually crashing it
partial a detector flagged a co-located dialog overflow and delivered it as a HIGH, but the exact site the public exploit crashes was missed by the extent heuristic
partial the root-cause site is detected and even built into candidate chains, but the solver skips this bug shape, so it is never delivered as a top prompt
n/r
Read that table with the header in mind. Only the first, the access-control client, reaches the operator's #1 prompt. The chat server is delivered end-to-end but ranked #26. The FTP case is shakier still: the detector flagged a nearby overflow and missed the exact site the public exploit crashes. The last is detected but never delivered to a prompt at all. And neither of the two genuine deliveries is a solver proof; both are pushed into the operator's queue by a post-processor, the honest inverse of the false positive below, which the solver actually did prove. Every one of these detectors was written against a binary whose bug was already public.
Synthetic: the harness, not the field
These numbers say the plumbing works. They say nothing about finding real bugs, and two of the three targets were written by us.
Corpus
What it is
Result
Time
Juliet C
a public synthetic memory-safety test set, 462 cases
synthetic 96.5% recall; use-after-free and double-free hold at 100% as the standing regression gate
n/r
MFC dialog
a Win32 dialog app we wrote ourselves
synthetic the real bug becomes prompt #1, on a target built to be found
n/r
.NET corpus
hand-authored C# samples, 29 cases
synthetic 100% pattern recall, which the project's own note is careful to call a measure of pattern-matching, not real precision
n/r
One that wasn't real
Kept here on purpose. The most dangerous output a tool like this produces is a wrong answer that looks submittable.
Case
Target
Claimed
Result
Time
the finding that wasn't
an industrial polling application
local-file code execution via a deserializer
false positive a HIGH-confidence, solver-proven prompt, refuted by the adversarial reviewer on reachability: dead-linked library code with 0 callers and 0 references
n/r
This one refutation is why a reachability filter exists at all. Turned loose on the backlog, it demoted 19 of 50 previously "Golden" prompts, and every one of the 19, checked by hand, was safe. The story is Post 10.