Results of improving the agent in an overfitting environment
Overview
The starting point for this test came from the following line of thinking.
The performance metrics we might call an LLM's "intelligence" have been trending upward with every model release.
I read that situation as: "IQ is going up, but expertise — or problem-solving ability — is 'not there yet.'"
At that point my thinking turned to: "How did I go about getting good exam scores back in school?" The method I recalled was 'establish the concepts, then work through past exam papers' → 'work the papers, then study the explanations for the ones you got wrong.'
So I assumed that rising LLM performance meant the concepts were being established with each new model, and on that assumption I planned to build only the environment for the second half — 'work past papers, then read the answer explanations.'
Experiment
I wanted to run two experiments.
First: with only shallow knowledge of Active Directory lateral movement, could an AD Lateral Movement Agent be developed?
Second: with only shallow knowledge of the smart contract audit field, could an AI Smart Contract Audit Agent be built?
I intended to run both, but the first ran into Claude's cyber guardrails repeatedly, so I dropped it and proceeded with the second only.
For 'agent development from a position of no knowledge' as I conceived it, two conditions are required: a playground containing the problems to be solved, and the existence of write-ups for that playground.
For example, to build a CTF pwnable player agent, you'd need a large number of wargame or CTF problems and the corresponding write-ups.
Given those conditions, I expected the system would end up chasing only what's in the 'answer key' — the write-ups — and that something resembling overfitting would occur. That was my hypothesis going in.
Smart Contract Audit Agent
The initial design was done in Claude's Plan mode.
The initial architecture was based on a multi-agent audit system I had previously designed with Gemini.
https://efficacious-collision-7ce.notion.site/Audit-Agent-System-4-Audit-Agent-Architecture-31c78187eb49805d8925e77dc0a6ef9f?source=copy_link
What I carried over from the old structure was only this: whether an orchestrator exists, a judge loop to filter false positives, writing PoCs on the basis of the test code already in the project, and the rule that vulnerability analysis must proceed from multiple perspectives.
After that I asked for a record schema to support post-solve improvement work.
The agent improvement process:
Run the audit → produce audit results while recording against the defined schema → compare results against the actual report from that audit contest → improve the agent structure based on the recorded material
Every audit run goes around this loop.
The final plan produced the following work tree and audit agent analysis flow.
auditforge/ # deployment package
├── SKILL.md # thin orchestrator (<300 lines)
├── VERSION
├── references/
│ ├── frozen/ # learning must not edit
│ │ ├── shared-rules.md # trust model · invalid catalog · rejection criteria
│ │ ├── platforms/
│ │ │ ├── code4rena.md
│ │ │ ├── sherlock.md
│ │ │ ├── codehawks.md
│ │ │ └── cantina.md
│ │ ├── poc-rules.md # inheritance pattern · cheatcode policy · assertion table
│ │ └── report-format.md
│ └── learned/ # learning may edit
│ ├── lenses/
│ │ ├── fund-flow.md
│ │ ├── ordering.md
│ │ ├── trust-boundary.md
│ │ ├── extreme-values.md
│ │ └── economic.md
│ ├── rejection-criteria.md # list of bad rejection reasons
│ ├── interaction-probes.md # graph-structure trigger questions
│ └── vuln-patterns/ # last resort. card cap in place
│ └── VP-*.yaml
├── schemas/
│ ├── finding.yaml
│ ├── trace.yaml
│ └── evidence.yaml
└── scripts/
├── probe_build.sh
├── validate_poc.py
├── slice.py # Phase 5
└── lint_cards.py
harness/ # not deployed
├── archive/
│ └── <platform>-<name>/
│ ├── repo/ # target snapshot (immutable)
│ ├── truth.yaml # normalized answer key
│ ├── meta.yaml # platform, date, rules_version, lang
│ └── baseline.json # score with knowledge emptied
├── graders/
├── gate/
└── runs/<timestamp>/
[0] Scope README parse → in-scope file list
↓
[1] Environment probe_build.sh → confirm forge build/test succeeds
↓ STOP on failure. Do not proceed to analysis
[2] Document reading DocumentReader → spec summary + trust model extraction
DocumentMatcher → check implementation against spec
↓
[3] Analysis unit split v1: file / contract level
Phase 5+: slice level
↓
[4] Parallel lens pass each unit × 5 lenses → hypothesis generation
hypothesis budget enforced, trace recorded
↓
[5] Gate sequence G1 Invalid-catalog → G2 Trust → G3 Reachability
→ G4 Impact → G5 PoC
↓
[6] PoC write/validate VulnTester → validate_poc.py → PROVEN/REFUTED/INVALID/ARGUED
↓
[7] Severity mapping evidence fields → platform grade
↓
[8] Submission threshold select top N
↓
[9] Report render output in platform format
The agent improvement process
Over 11 days I ran 41 scored rounds.
This section records only how the improvement environment ran and what came out of it.
Each stage is the result of running again after improving on the previous stage's failure.
Before you read — terminology
First, who did what in this write-up. The improvements below were not done by my hand. I built the environment and watched the results; the one that actually ran the rounds, diagnosed the failures and fixed the structure was the orchestrator.
| Actor |
What it did |
| Me (the author) |
Initial architecture design, record schema definition, setting up an environment with problems and answer keys. After that, watched the loop run and wrote this post |
| Orchestrator |
Executes rounds, scores against the answer key, diagnoses failures, fixes the pipeline structure and scripts. The de facto agent of the improvement loop |
| Sub-agents |
Execution units spawned by the orchestrator. Perform the actual analysis, refutation and recording |
This distinction matters later. Sections 13 and 14 draw their conclusions precisely from the point that "the orchestrator scored and diagnosed itself."
And here is the notation and the set of parts that recur below.
| Notation |
Meaning |
| R1, R2 … R27 |
Round numbers. One round is the full cycle of "attach the pipeline to one past contest and audit it → score it against the answer key → look at the record and fix the structure." The number is execution order. |
| C1, C10 |
Correction numbers. Serial numbers attached to measurements the orchestrator produced and then invalidated itself. They were kept and numbered rather than deleted. |
| Part |
What it does |
| Lens |
A single perspective. Early on it was a prose instruction like "look at this from a fund-flow perspective"; later it became an assertion — "this code is safe" — plus a list of points at which to attack it. That change is half of this write-up. |
| Analysis unit |
A piece of the codebase. One entry point bundled with the code that function actually reaches. The unit of work one agent holds at a time. |
| Detector |
A script rule that sweeps the source and emits a located question. It never says "this is a defect" — only "ask about this spot in this way." |
| Gate |
A sequential check a finding must pass to reach submission. Which gate it failed at becomes the input to failure diagnosis later. |
| Metric |
Meaning |
| Recall |
The proportion of that contest's confirmed answers we pointed at. 5/8 means 5 of 8. |
| Award value |
The value of one problem. Computed by the contest formula; the more people who found it, the closer to zero. |
| Rank |
Our position when inserted among the people who actually competed in that contest. |
0. Just running it — it was dismal
I attached the designed pipeline to three past contests. As a control I ran a plain pass with no perspective lenses alongside it, to see whether the lenses were worth anything.
| Round |
Plain pass |
Lens pass |
Rank |
| R1 |
2/3 |
2/3 |
11 / 239 |
| R2 |
1/4 |
1/4 |
11 / 58 |
| R3 |
0/8 |
0/8 |
83 / 83 |
R3 was 83rd out of 83 participants — dead last.
But there was something worse. For three rounds running, the lenses added exactly zero. I attached a prose instruction saying "look at it from this perspective," and the agent read it, said "yes, I'll look at it from that perspective," and then looked at what it was going to look at anyway.
The R2 record put it this way — "the cause is not the lenses, it's the absence of input."
1. First realization — the metric itself was wrong
I looked at R1 again. It got 2 of 3 right and scored 0.0000.
This contest's award formula is:
award = weight × 0.9^(n-1) / n
n is the number of participants who actually found that problem in the contest.
A problem many people found converges to zero.
Laid out as a table, it's a cliff.
finders n high-severity award medium-severity award
1 10.0000 3.0000
2 4.5000 1.3500
5 1.3122 0.3937
8 0.5979 0.1794 ← the problem R1 missed
10 0.3874 0.1162
25 0.0319 0.0096
77 0.0000 0.0000 ← a problem R1 got
207 0.0000 0.0000 ← a problem R1 got
The two the pipeline got right were also found by 207 and 77 of 238 participants.
That contest's entire value sat in one problem that only 8 people found, and it missed that one.
"How many did you get right" was not a performance metric.
In exam terms, you don't look at the percentage correct — you look at whether you got the high-value problems.
The scoring script was rewritten to reconstruct that contest's actual leaderboard.
From this point on, recall and rank are always read together. That decision became the baseline for every judgment that followed.
2. Prose instructions became detectors
Since the diagnosis for the zero lens contribution was "prose doesn't make the question land," the prose was turned into code. What came out of that is the detector.
The core design is this:
What a detector emits is a question, not a finding.
It's written into the deployed skill document verbatim —
*"A hit is a question, never a finding."
A detector never says "this shape is a defect." Most of the time that shape is perfectly ordinary code. All a detector does is point at a location and emit a question — things like "who consumes this value later?" or "what happens if the state changes after this check passes?"
This is where the "answer explanation" loop actually started turning.
run a round → score → look at the answers that were missed →
work backwards to "what question would have found this?" → freeze it as a detector
The detectors grew to 26 kinds. Recall came up into the 10–30% range.
And then it stopped there.
3. "So let's make it read all the code" — pushing coverage
The next hypothesis was simple: it doesn't find things because it doesn't read them.
Coverage was adopted as a metric and pushed for several rounds. The logic for extracting analysis units was fixed, budget rules were changed, and code types the walker structurally could not see were folded into enumeration.
contract-level coverage 53.0% ████████████ → 86.5% ████████████████████
function-level coverage 15.0% ███ → 66.0% ███████████████
answers inside the analysis set 29.0% ██████ → 93.0% █████████████████████
The numbers went up nicely. And then the next round produced this.
4. Coverage went up and recall didn't
I ran a round with coverage held as a fixed variable.
code taken as the analysis set 22 of 23 units = 95.7%
rows actually filled on top of it 1,577
that contest's answers 26
coverage ███████████████████████████████████████████████ 95.7%
recall 0.0%
precision 0.0% (0/4)
95.7% of the code was in the analysis set and it scored 0%.
I opened the misses one by one. Every one of them was a case where the file containing the answer was in the analysis set and the agent had opened that file — and still missed it.
Coverage was a necessary condition, not a sufficient one.
The code was read; the question simply never landed.
Until this round coverage had been used as a proxy for performance, and it was discarded here. Around the same time the coverage report itself was corrected — "not selected" and "nobody looked at it" are different events and were being counted as one number.
5. So what is the bottleneck — it was depth, not breadth
Because the previous round held coverage as a fixed variable, for the first time something else separated out as the bottleneck.
Two things have to be distinguished. Until this point the loop had been conflating them.
|
What it measures |
How it's counted |
| Breadth (coverage) |
Whether code containing an answer went into the analysis set |
Proportion of all code taken as a target |
| Depth |
How far into that target it dug |
Number of units actually analysed, rows of record filled |
The previous round was 95.7% breadth at 1,577 rows of depth. A wide sweep read shallowly.
So on the same target, same lenses, same analysis-unit extractor, same model, I changed depth only. All I did was raise the number of concurrently running agents from 1 to 12.
|
1 agent |
12 agents |
| units actually analysed |
20 |
79 |
| rows of record filled |
2,729 |
12,024 |
| recall |
0/10 |
7/10 |
0% → 70%. Not one line of methodology changed. It looked at the same code four times as thickly.
Confirmation in the same direction came three times.
| Comparison |
What changed |
Recall |
| Target A |
20 units → 79 units |
0% → 70% |
| Target B |
20 units → all of them |
67% → 87% |
| Target C |
(see below) |
100% where analysed / 9% where not |
Target C works differently. Instead of changing the execution, I took the result of a single run and split the answer locations into two groups.
Splitting that contest's 14 answers by whether they sat inside
code the agent actually analysed —
answers inside analysed code 3 of 3 found = 100%
answers outside analysed code 1 of 11 found = 9%
Where the analysis reached, it found nearly everything; where it didn't reach, it found nearly nothing.
Put another way, that round's failure was not a judgment problem but a volume problem.
The rule that came out of this:
Split the analysis units first, perspectives second.
Six lenses independently arrived at the same defect redundantly, whereas raising the number of agents produced new defects. Seeing more code from one angle was worth more than seeing the same code from six.
6. The misses that remained had a different shape
Even with thick analysis, misses remained. Classifying the failure shapes, most were one thing.
"Reached the right location and asked the wrong question."
And there was a worse variant — finding it and then rejecting it yourself.
| Round |
What happened |
Award value of that problem |
| R9 |
Read the answer location, cited it, and asked about something else |
1.3500 |
| R10 |
Read the code in question and classified it as "normal" |
0.1162 |
| R24 |
Acknowledged the defect then demoted it as "unlikely," never filed |
3.0000 |
| R27 |
Four agents read that location and none of them filed |
0.5468 |
| — |
Three analysts independently reached the same defect, all three rejected it |
— |
R24 was especially important. The agent wrote this in its own record:
real defect but requires [condition]; noted, not asserted (Low)
From the answer key you can only see "it was missed."
You have to read the output to see "it acknowledged it and demoted it itself."
The rule that came out of this is "likelihood is an argument about severity, not an argument about whether to file."
7. The refutation pass — recovering without reading more code
Collect only the items the pipeline concluded were "fine."
Then hand the claim and the location only to an agent that has never seen that conclusion.
Additional code read: zero.
Someone else audited this code and concluded it was fine.
You are being given their CONCLUSION and the file paths —
never their reasoning, and never the finding they wrote.
Independence is the mechanism. Re-reading your own conclusion reproduces that conclusion.
Aggregate across 15 runs over seven targets:
conclusion broken 12 / 15 ████████████████████████
conclusion held 3 / 15 ██████
of those, became confirmed answers 5
the 3 that held paid off too — all three replaced an
assumption with verified evidence rather than 'agreeing'
Recall went 70% → 90%. It reproduced across three rounds.
And the orchestrator invalidated all of these numbers itself. That is correction C1.
correction: all refutation pass numbers void — I picked the targets having seen the answer key
The orchestrator picked which targets to re-attack, and at that moment the orchestrator had already seen the answer key. Scoring requires opening the answer key, and the same actor designs the next experiment.
The "I" in that commit is not the author but the orchestrator that ran and scored the pipeline. This structure leads to the conclusion in section 14.
The rule changed afterwards — take target selection away from the human and feed everything in unfiltered.
If someone picks the targets, that pass's ceiling is whatever the picker already suspected.
Only one round was re-measured cleanly, but that one was large.
A refutation pass run blind caught an award value of 10.0000 — a problem only one participant in the entire contest found. It is the single most valuable item in this whole experiment.
By recall it's 1/6 = 17%; by value it's the best.
8. Refutation expanded to three directions
Once one direction worked, I opened others. Each came from a different failure shape.
① Re-attacking rejections — an independent agent hits the "it's fine" conclusion again. (The one above.)
② Finding the other victim — the name can be confusing, so spelled out:
When an agent finds a defect, it names one person who is harmed and files on that. But the same line of code often harms a completely different party too, and what the judges accepted was often that other party's side.
A real case makes it fast:
What was found a permission check misfires, so the admin cannot change a setting
Victim filed the admin → severity: "operational inconvenience"
Asked "who else is harmed?" —
Second victim ordinary users → with that setting unreachable, deposit processing
fails permanently, and money already sent in is
stuck having received 0 shares. No cancel path.
Same single line of code, and it goes from "admin inconvenience" to "user fund loss."
What was found was right from the start; only the victim was misidentified.
There was a round where rank turned on exactly this shape. A round concluded "the check doesn't run on path X, so X is unaffected," while the confirmed answer drew the opposite conclusion from the identical sentence — a path where the check doesn't run is a path usable without any constraint. Same fact, same location, opposite conclusion. That was the difference between 89th and 6th.
So every accepted finding is now forced to answer "who else does this fact make short?"
③ Re-reading with location only — the first two passes share a weakness. Both hand the agent a conclusion while telling it to attack. ① tells it "someone concluded this is safe here," ② tells it "this defect was found here."
Reading the outputs showed the problem. Even when told to attack, that conclusion becomes the starting point of thought. It's the same as telling a person "prove this sentence is false" — they end up thinking around that sentence. They don't look at the code afresh; they go to check whether the given sentence holds.
So the third pass gives no context at all.
What it gets file path and line number. That's it.
What it doesn't what was found earlier
what was rejected earlier
why that line was cited
who said it was safe
Question "how can an unprivileged user use this line?"
Only the list of line numbers the earlier rounds cited is extracted and handed to an agent seeing those lines for the first time. That agent has no idea what was said about this code earlier.
The actual output of the three passes in the most recent round:
| Pass |
Input |
Result |
| ① Re-attack rejections |
15 safety conclusions |
12 broke |
| ② Find the other victim |
29 accepted defects |
all 29 had another victim |
| ③ Re-read with location only |
154 cited locations |
13 read the opposite way |
② returned zero cases of "no other victim." All 29 filed items had a party harmed beyond the one originally named.
And ③ caught 2 defects that all 13 lenses had missed — lines every earlier round had read as "normal" and passed over.
9. The last ceiling — removing my own influence
At first there were imperative constructions like "find the vulnerabilities."
But as the agent improved automatically each round, it changed to this:
|
Before |
After |
| What the agent is given |
"find the defects" |
the assertion "this code is safe" |
| The agent's job |
open-ended search |
refutation of that assertion |
| Output |
what it found |
broken (defect) or holds (evidence ledger) |
When the thing to be refuted is stated, the questions you didn't ask become visible.
With "bring me anything" you can't tell what's missing; with "this is safe" you end up looking for the vulnerability out of contrarian instinct.
The result of handing the next round the assertion "this code is safe" and having it solve from there: 16 of 25 confirmed answers, 1st out of 249 participants.
10. The odd part — there isn't a single 'vulnerability pattern' card here
Look at other published audit agents or skills and most of them stand on a vulnerability pattern list. This shape is that defect, that shape is this defect — put the list in the prompt and have it match against them.
My initial design had a slot for that too. It's references/learned/vuln-patterns/ in the tree above, and the design document says "last resort. card cap in place."
Now that 41 rounds are done, that directory has zero cards.
The orchestrator never made a single one. And that isn't laziness — it's a measurement result. It's written into the deployed skill document:
Do not preload `vuln-patterns/`: preloading measurably hurts hits.
Injecting patterns in advance lowered the hit rate.
The reason is clear in hindsight. Give an agent patterns and it goes hunting for those patterns. It finds what's on the list, and doesn't look at what isn't. But the high-value problems in a contest are by definition rare, and rare means not on a list of common patterns.
The most valuable finding in this experiment — award value 10.0000, a problem only one participant in the entire contest found — is on no pattern list anywhere. If it were, only one person wouldn't have found it.
What this system uses instead of patterns is structure.
|
Pattern-based |
This system |
| Input |
"this shape is a defect" |
the assertion "this code is safe" |
| What it does |
match against a list |
refute that assertion |
| What it can't find |
anything not on the list |
(there is no list) |
The 26 detectors aren't patterns either. There's a decisive difference.
A detector never says "this is a defect."
It asks "who consumes this result?"
Three detectors fire at locations that had already been read and passed. One of them fires on a spot where a round had traced every path consuming that point and concluded it was "correct design." That round asked in one direction only and never once asked the reverse.
Patterns tell you "what is a defect"; this approach tells you "what you didn't ask." The latter aims squarely at the failure shape from section 6 — "reached the right location and asked the wrong question."
While recall rose 18.2% → 56.9%, and while an external benchmark returned 65%, not one pattern card was added.
This doesn't mean "patterns are useless," of course. No A/B was ever run with cards loaded, so the effect of adding patterns was never measured. Only one thing was measured — preloading lowers the hit rate. So the skill doesn't ban cards; it leaves them to be requested by ID when the operator needs one for a question. It's just that in 41 rounds that request never once came.
11. The flow, summarized
Each stage came out of the previous stage's failure.
just ran it
→ last place (0/8, 83rd/83), perspective contribution zero three times running
↓ "prose instructions don't make the question land"
switched to detectors
→ stalled at 10–30% recall
↓ "it doesn't find things because it doesn't read them"
pushed coverage (53% → 86.5%, function 15% → 66%)
→ 0% recall at 95.7% coverage
↓ "the code was read. the question didn't land"
made the analysis thick (1 agent → 12, 20 units → 79)
→ 0% → 70%
↓ "the remaining misses are things found and then rejected"
refutation pass (reads no additional code)
→ recovered. but numbers void — the orchestrator picked the targets (C1)
↓ "take target selection away from the human" + expand directions
find the other victim · re-read with location only
→ caught defects all 13 perspectives had missed
↓ "the gap that remains = the orchestrator wrote the claims"
claim authoring moved into a script — recast as refutation of "this code is safe"
→ 16 of 25, 1st of 249
The performance trend. (The axis is unified to the same platform and the same language.)
| Period |
Rounds |
Aggregate recall |
Rounds scoring 0.0000 |
| Formation (R1–R18) |
15 |
10/55 = 18.2% |
8 / 15 |
| Pre-registration + deployed build (R19–) |
8 |
70/123 = 56.9% |
0 / 7 |
The right-hand column is sharper than recall. In the formation period more than half earned nothing at all.
Laid out by round:
Formation ────────────────────────── Pre-registration + deployed ─────────
R1 ████████████████ 67% (2/3) R19 ██████████ 40% (6/15) 4th/117
R2 ██████ 25% (1/4) R20 ████████████ 48% (11/23) 4th/185
R3 · 0% (0/8) last R21 ████████████████ 64% (9/14) 1st/147
R5 ████████ 33% (1/3) R23 ████████████████ 64% (16/25) 1st/249
R6 ██████ 25% (1/4) 6th/67 R24 ████████████████ 64% (7/11) 2nd/166
R9 ██████ 25% (1/4) R26 ██████████ 43% (3/7) 20th/73
R13 · 0% (0/2) last R27 ███████████████ 63% (5/8) 17th/83
R16 · 0% (0/6) last ────────────────────────────────────
R18 · 0% (0/4) last external ████████████████ 65% (13/20)
Last place came up five times in the formation period. And the five rounds after R23 are 64 · 64 · 43 · 62 · 65 — flat. One step up, then a plateau.
There is one comparison from which causation can be claimed.
R3 and R27 are the same contest, the same answer key, the same scoring script.
R3 (day 3) → 0/8 · 83rd / 83 (last)
R27 (day 5) → 5/8 · 17th / 83
The confounder is on the record too. The R3 notes say it at the time — the R3 target was far larger and the context budget ran out, so it walked shallowly. Part of that zero is depth, not method. This comparison measures not "the improvement" but "the improvement plus the execution headroom."
Conclusion
Going in, I thought this environment — studying against answer keys — would produce overfitting, and the improvement direction I predicted was that it would learn "vuln patterns."
After improving on the results of 41 audits, what it did instead was attempt to raise coverage, revise the analysis perspective, revise the gates to reduce false positives — it developed itself not toward learning vuln patterns but toward "what question do I have to ask in order to find the vulnerability," and recall has trended upward so far.
The remaining limitation, though, is that the probability of finding vulnerabilities that few people found — that is, the high-award-value vulnerabilities — is still not high.