Why Your AI Agent Can't Solve CTF Challenges: Pwnable

An AI agent can find a vulnerability and write exploit code yet still fail to capture the flag. It might overwrite a function pointer without obtaining the address to put there, or misdiagnose a crash and repeat the same input. Generating code leaves further work: checking the conditions the exploit needs and correcting failures.

There are also successful flag captures, but improvements from changing the model need to be distinguished from those achieved by changing tools and execution procedures. Following the reverse engineering and web hacking installments, this article examines where information gathering, exploit design, execution, and validation break down in pwn challenges.

How Models and Harnesses Affect Solving Performance
Missing Analysis Inputs and Tool Observations
The Gap Between Finding a Vulnerability and Building a Working Exploit
Misreading Execution State and Revising Inputs
Diagnosing Failures and Deciding Where to Retry
Defining Success and Validating Exploit Behavior
Conclusion
References

How Models and Harnesses Affect Solving Performance

Model capability alone is not enough to judge an agent's ability to solve pwn challenges. Even with the same model, results depend on whether the agent can call the tools it needs and use execution results to guide its next attempt. A harness is the execution setup that manages these tool calls, returns their results, maintains working records, and handles retries. It can also delegate tasks to agents specializing in analysis or debugging and collect their results.

The researchers behind Pwnagent, an automatic exploit generation system, compared configurations with and without dynamic measurement and diagnosis-guided iterative repair while keeping the model fixed. Pwnagent assigns analysis, dynamic measurement, and code generation to specialist agents and passes GDB measurements to the code generation stage. When execution fails, the diagnosis guides a review of measurements, implementation, or strategy. The comparison configuration retained static analysis and agent role specialization but removed both dynamic measurement and diagnosis-guided iterative repair, generating code in a single attempt [1].

The overall architecture of Pwnagent (original paper, Fig. 2, p.7). Stage 3 measures values at runtime; Stage 5 diagnoses failures and returns to earlier stages for repair. The success rates discussed here compare a configuration using both capabilities with one that generates code once without either capability [1].

With Kimi-K2.6 used in both configurations on 66 local CTF challenges, the rate of obtaining a shell or reading the flag was 62.12% (41 challenges) for the full configuration and 42.42% (28 challenges) with both capabilities removed. This difference reflects the combined effect of providing dynamic measurement and iterative repair [1].

Pwnagent records analysis results, strategies, and measurements in a central state and passes them to subsequent agents. Its evaluation sample is concentrated on stack-oriented challenges and x86 architectures [1].

The researchers behind the CTF agent EnIGMA also compared tool interfaces while keeping the model fixed. On 44 pwn challenges using Claude 3.5 Sonnet, the full configuration captured flags in 20.45% of challenges, compared with 11.36% when both the GDB and remote connection interfaces were removed. The evaluation used a single run with a $3 budget per challenge, allowing multiple flag submissions within that run [4].

EnIGMA is an agent used to study tool interfaces for CTF solving. It provides interfaces for starting and stopping interactive tools and sending commands to them [4].

ExploitGym provides a comparison of model changes. It evaluates whether an agent can turn a proof-of-vulnerability (PoV) input into an exploit that captures a flag, comparing GPT-5.4 and GPT-5.5 with the same Codex CLI 0.120.0 and xhigh reasoning effort. Across 898 tasks covering userspace programs, V8, and the Linux kernel, successful flag captures using the specified vulnerability increased from 54 to 120. Each task allowed 1 attempt of up to 2 hours under trusted-access conditions, with security mitigations and deployment-time safety filters disabled [2].

ExploitGym is a benchmark that evaluates vulnerabilities in real software within containerized environments. Beyond reproducing a vulnerability, it checks whether the agent captured the flag in the target environment and used the specified vulnerability [2].

These comparisons show that solving performance can change with the harness when the model is held fixed, and with the model when the harness is held fixed. An agent's performance is the result of a particular model-and-harness combination. Reading a success rate therefore requires looking beyond the model name to the tools, task allocation, retry procedures, and evaluation conditions.

Missing Analysis Inputs and Tool Observations

An agent can keep revising code and still make no progress if essential information is missing from its inputs. Binary analysis may overlook a library, or the agent may be unable to query a value available only at runtime. The question is whether analysis results and tool outputs actually deliver the information the agent needs.

In the evaluation of PwnGPT, which passes binary analysis results to exploit generation, qwen-plus, qwen-max, and GPT-4o all failed to gather the necessary information for three challenges. One was rop-10, a challenge involving a stack overflow in a 64-bit ELF binary, intended to be solved with ret2libc: redirecting execution to libc functions. The analysis module did not process dynamically linked libraries, so it omitted libc gadgets—short instruction sequences that can be combined in an exploit—and could not assemble the inputs needed for exploit generation [3].

PwnGPT is an automatic exploit generation framework with analysis, generation, and verification modules. Its analysis module selects key functions and information from decompiled output to construct the generation input [3].

When information is available only during execution, the way the agent queries it matters as well as the scope of analysis. EnIGMA focused on shell-centered agents that did not support separate interactive sessions. Debugging often requires examining memory before deciding what to inspect or execute next, making it difficult to gather all the necessary observations with a script written entirely in advance. EnIGMA opens GDB and pwntools remote connections in separate interactive sessions, letting the agent send further commands after receiving a response. The main shell remains available while one interactive session is active [4].

Before generating code again, the agent needs to change its information-gathering method to address the specific omission. If libc was left out of the analysis, as in rop-10, analyzing that library and including the required gadgets can fill the input gap. If runtime addresses or state are needed, an interactive tool such as those provided by EnIGMA can query and return them. Code revision should follow a check that the information gathered has actually reached the agent's input.

The Gap Between Finding a Vulnerability and Building a Working Exploit

Finding a vulnerability or gaining the ability to manipulate memory does not establish every condition needed for the next step. Overwriting a function pointer is useful only if there is a valid execution address to put there. Attempting the final operation before securing these intermediate conditions can leave an exploit unable to work as intended, even when the vulnerability was identified correctly.

Where Do LLM Agents Fail in Binary Exploitation? studied agent failures on 28 public CTF challenges, allowing each agent–challenge pair up to 3 attempts of 60 minutes each. C5, a challenge identifier in the study, was a QuickJS use-after-free (UAF) challenge compiled as a position-independent executable (PIE), a build configuration that allows ASLR to randomize the executable's code base address. UAF means accessing memory after it has been freed. Codex CLI/GPT-5.3 overwrote a function pointer but skipped the address leak. Without valid addresses, the payload crashed, and the agent exhausted its time budget redesigning the exploit [5].

Where Do LLM Agents Fail in Binary Exploitation? is an IEEE S&P 2026 poster study that analyzes failure points using the last stage completed in execution logs [5].

What C5 lacked after vulnerability discovery was an address needed for actual code execution. To distinguish such gaps, the researchers classified the last completed state in the execution logs into five categories [5].

Vulnerability discovery: Find the vulnerable code and the input conditions that reach it.
PoC verification: Check that the input or code reproducing the vulnerable behavior works.
Address leak: Obtain the runtime addresses needed by the exploit.
Primitive construction: Build basic capabilities for memory reads, writes, or control-flow manipulation.
Goal achievement: Use those capabilities to reach the final objective. In this study, that meant capturing the flag.

The required stages and their order depend on the solution. To reduce skipped intermediate steps, the researchers proposed explicit, verifiable subgoals, requiring each to succeed before proceeding. Applied to C5, this means separately verifying the function pointer overwrite and the acquisition of a valid address to place in that pointer [5].

A separate ExploitGym evaluation illustrates how intermediate capabilities can lead to flag capture. GPT-5.4 modified an input that crashed only a debug build of V8 to create a read primitive in a release build. It validated that primitive, leaked pointers and a libc address, and connected these capabilities to code execution and flag capture in the target container. This success occurred in d8, the standalone V8 shell, with security mitigations disabled. When ASLR, which randomizes address placement, and the V8 heap/renderer sandboxes were restored, code execution failed [2].

Exploit construction requires breaking the prerequisites for the next operation—such as obtaining an address or building a primitive—into subgoals and checking each under the current environment. If a prerequisite is missing, it needs to be established and connected to the next operation before the agent repeats the final payload.

Misreading Execution State and Revising Inputs

Access to execution state does not prevent repeated payload failures unless the agent uses those observations to correct its assumptions. The actual memory location may differ from the expected one, or execution may never have reached code the agent assumed it had passed.

The Pwnagent researchers illustrated this with the index needed to overwrite a return address stored on the stack in the CTF challenge stack2. Assuming a conventional stack frame gives an index of 116, but the actual value was 132 because the function restored a saved stack pointer as it returned. By examining the restoration path in the assembly and measuring the return address's location with GDB, an agent can recalculate the index for the actual location [1].

The expected layout (a) and actual layout (b) in stack2. A conventional frame assumption gives an index of 116, while accounting for stack pointer restoration gives the actual index of 132. The restoration path is also visible in the assembly. Source: Chaojie Wei et al., Pwnagent, Fig. 1, p.4 [1]. CC BY 4.0. Only the figure region was extracted; its labels and layout are unchanged.

An input's execution path can differ from expectations just as memory locations can. DrillAgent, which generates PoVs for C/C++ programs with source code available, maps coverage—the code actually executed—to source locations to determine how far an input has progressed. The agent examines the conditions on paths that fail to reach the target and revises the input. Inputs that reach the vulnerable function are preserved and reused while the agent works out the conditions needed to trigger a crash [6].

To evaluate this feedback, the researchers selected 30 tasks that DrillAgent had already solved, prioritizing cases that other methods had not solved. With Sonnet 4.5 as the main model and a fixed assignment of models to subagents, the full configuration reproduced all 30, while removing coverage feedback reduced the count to 17. Success meant producing a PoV whose sanitizer-reported crash matched both the target crash type and location. This evaluation did not measure flag capture [6].

DrillAgent is a PoV generation system that combines vulnerability root-cause analysis with execution-path observations. It reports whether functions were reached and provides tools for querying coverage of specific source lines or regions [6].

Before revising an input, the agent needs to identify where its expectations about memory locations or execution paths differ from the observed run. An unexpected return address location calls for reviewing the index calculation; failure to reach the target code calls for reviewing the input conditions along that path. After changing the input, another execution should confirm whether the discrepancy has been resolved.

Diagnosing Failures and Deciding Where to Retry

A retry requires deciding where to resume as well as how to revise the input. If earlier operations corrupted memory state, changing only the code that failed last can reproduce the same failure. Discarding every validated result, on the other hand, repeats preparation that was already complete. The problem is deciding which earlier results to retain and which assumptions to revalidate.

In the failure study discussed earlier, C13, another challenge identifier, involved a heap UAF race condition. Claude Code/Opus 4.6 triggered the UAF, but doing so corrupted the free-list—the allocator's record of freed memory blocks—and destabilized subsequent heap operations. Even as address leak and arbitrary write attempts failed, the agent did not recognize this cause and kept retrying the same approach [5].

Choosing a restart point first requires identifying which earlier work remains valid. Refploit, a system for repairing Java library exploit reproductions, makes this assessment separately for environment preparation, invocation of the vulnerable API, and reproduction of the intended behavior. It tells the recovery agent which environment and capabilities to preserve, which requirements to repair, and which misleading paths to avoid. It replays a validated initial segment before resuming unfinished work; if no reliable segment exists, it starts from the initial workspace state [7].

Refploit analyzes execution records to repair exploit reproductions. It reruns the modified artifacts against vulnerable and patched versions, checks the difference, and reassesses whether recovery succeeded [7].

In an OpenRefine case repaired by Refploit, the project setup and MySQL/JDBC direction were reusable, but the initial code simulated the behavior with its own implementation instead of using the actual vulnerable library. Refploit preserved the preparation, replaced the incorrect implementation with the real library and vulnerable API call, and revised the payload. It confirmed successful reproduction by checking that reading the target file succeeded on the vulnerable version but was blocked on the patched version [7].

Failing to recognize completed work can also stop a repair prematurely. In Refploit's XStream case, an exception raised after reaching a deserialization path was classified as an invalid attempt, and payload adaptation stopped. The researchers described this as a failure to recognize partial progress that still needed further adaptation [7].

Retries need separate records of confirmed behavior and the point where an error occurred, so the agent can decide what to preserve and which assumptions to revalidate. Triggering a UAF, as in C13, does not establish that subsequent heap operations have the state they require. Corrupted state must be brought back to a valid state, and unfinished work should resume after replaying the validated preceding steps.

Defining Success and Validating Exploit Behavior

Interpreting these results as evidence of solving ability requires checking what each evaluation counts as success. Writing code, reproducing vulnerable behavior, validating a primitive, and capturing a flag are different outcomes. Even a flag capture has different implications depending on whether the agent read an external answer, obtained it from the target, or used the specified vulnerability.

The CTF evaluation study CTFusion reported that adding web search allowed agents to copy external flags or use public write-ups, making it difficult to evaluate their ability to solve challenges directly. In 1nsayne, a reversing challenge in NYU CTF Bench, the agent installed nyuctf, a Python package providing access to the benchmark's challenge dataset, and read the flag from it. It obtained the answer without reproducing the required behavior in the target program [8].

CTFusion is an evaluation framework that uses unpublished challenges from ongoing CTF competitions. It investigates the limitations of static benchmarks with publicly accessible answers and evaluates agents in live competition environments [8].

When submitting the correct answer is not sufficient evidence of solving ability, the evaluation needs to inspect what actually happened in the target. ExploitBench evaluates 41 V8 vulnerabilities, providing a vulnerable build and a patch, and directly checks capabilities from reaching and reproducing a vulnerability to memory manipulation and control-flow redirection. Depending on the capability level, it compares behavior before and after the patch or validates it under randomized heap layouts, filtering out results that rely on success messages or fixed addresses. Its validation scope is the V8 process [9].

ExploitBench automatically grades actual exploit behavior and also inspects execution records for attempts to circumvent the evaluation [9].

Even after confirming actual behavior, evaluating the exploitability of a specific vulnerability requires determining whether that vulnerability was used. In the ExploitGym evaluation of GPT-5.5 discussed earlier, the agent captured flags in 210 tasks, but only 120 were counted as successes using the specified vulnerability. Runs using other vulnerabilities, among others, were excluded from the latter count [2]. A solution that could be valid in a general CTF may therefore be excluded from an evaluation of a particular vulnerability's exploitability.

An evaluation should first confirm that the target behavior occurred, then determine whether the route used to obtain that result fits the evaluation's purpose. PoV validation checks reproduction of vulnerable behavior; primitive validation checks the intended read, write, or control capability; CTF solving checks a flag obtained from the actual target. Evaluations of a specific vulnerability's exploitability also check which vulnerability was used. Keeping these criteria separate prevents code generation or partial progress from being mistaken for final success, while allowing flag captures through other vulnerabilities to be accepted or excluded according to the evaluation's purpose.

Conclusion

A pwn agent's performance depends on its model and harness, but repairing a failed run also requires identifying the work at which it stalled. Missing input information, skipped prerequisites, and misread execution state or success criteria call for different responses. The failure points discussed above suggest the following checks.

Failure point	Response	What to verify
Missing analysis inputs	Obtain relevant functions and libraries; support interactive observation	Did the required information actually reach the agent's input?
Skipped solution steps	Validate the required addresses and primitives separately	Is the next step possible under the current security mitigations?
Misread execution state	Compare expected and observed state, then revise the input	Which calculation or input changed because of the observation?
Misdiagnosed failure	Preserve validated results and choose a restart point	What should be retained, and where should revalidation begin?
Confused success criteria	Check the target behavior and the route used to obtain it	Was reproduction, a primitive, or flag capture actually validated?

Identifying what was missed and where it was missed determines whether to change the model, tools, records, or validation. Missing a prerequisite or misreading an execution result can prevent an agent from reaching the next step—and that is why your AI agent can't solve CTF pwn challenges.

References

[1] Chaojie Wei et al., Pwnagent: a knowledge-guided multi-agent system for automatic exploit generation, Cybersecurity 9:224, 2026. Official paper.

[2] Zhun Wang et al., ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, arXiv:2605.11086v1, 2026-05-11. Paper.

[3] Wanzong Peng et al., PwnGPT: Automatic Exploit Generation Based on Large Language Models, ACL 2025. Official paper.

[4] Talor Abramovich et al., EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities, ICML 2025, PMLR 267. Official paper.

[5] Yudai Fujiwara, Ryutaro Nishizaka, Yuichi Sugiyama, Poster: Where Do LLM Agents Fail in Binary Exploitation?, IEEE S&P 2026. Official poster paper.

[6] Haoyu Li et al., Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generation, arXiv:2602.13574v1, 2026-02-14. Paper.

[7] Zirui Chen et al., Refploit: Facilitating Exploit Construction via Code-Agent Trajectory Repair, arXiv:2607.01760v1, 2026-07-02. Paper.

[8] Dongjun Lee, Ga-eun Bae, Insu Yun, CTFusion: A CTF Based Benchmark for Evaluating LLM Agents via MCP, arXiv:2605.11504v2, 2026-07-11. Paper.

[9] Anthropic Frontier Red Team, Measuring LLMs’ ability to develop exploits, 2026-05-22. Official report.