Why Your AI Agent Can't Solve CTF Challenges: Reverse Engineering
Why Your AI Agent Can't Solve CTF Challenges: Reverse Engineering
AI agents have become indispensable for competing at the highest level across CTF categories. Challenges that once kept a human busy for days can now be handed off to an agent and solved with little more than a click. The same pattern is emerging everywhere—web, pwn, reverse engineering, cryptography, and forensics.
The problem is that there is a clear divide between the challenges agents solve well and those they do not. On some challenges, an agent finds a path forward before a human does. On others, it runs tool after tool but never clears the final hurdle. This gap is not unique to any one category. The details vary, but agents quickly lose their footing whenever they misread analysis output, lose context, or cling to a hypothesis without validating it.
This article is part of a series exploring these differences across CTF categories. The first installment focuses on reverse engineering. Drawing on conference talks, open-source tools, and papers published between 2025 and 2026, it examines how agents solve reversing challenges, the conditions under which they perform well, and where they struggle.
Table of Contents
How Agents Approach Reverse Engineering Challenges
An agent's basic strategy is to rapidly repeat the same analysis process a human reverse engineer would follow. It first checks the file format and protection mechanisms, then gathers static analysis results with tools such as strings, readelf, objdump, radare2, Ghidra, and IDA. When needed, it also uses GDB or tracing tools. From there, it forms a hypothesis around suspicious functions or strings, writes a Python solver or keygen, and tests it against the actual binary.
CrackMeBench's evaluation criteria show that, in agent-driven reverse engineering, a working result matters more than an explanation [2]. An agent's advantage lies not in choosing a fundamentally different direction, but in speed and repetition. A human might read the decompiler output, write one solver, and pause when it fails. An agent can test dozens of input variations without tiring, and it can write each new script quickly.
According to the CREBench evaluation, models are relatively good at recognizing patterns commonly found in standard cryptographic implementations [13]. The same applies to CTF reversing. Agents quickly solve challenges whose validation logic is easy to see in static analysis, including constant comparisons, simple encodings, string transformations, straightforward keygens, and well-known cryptographic patterns.
CrackMeBench is a benchmark designed specifically for reverse engineering. Instead of grading free-form explanations, it externally evaluates inputs, serials, artifacts, or keygens that the target program accepts. Agents receive binaries with little to no symbol information and a standard reverse engineering toolset, then solve the challenges inside a network-isolated Linux Docker environment [2].
CREBench is a benchmark consisting of 432 challenges at varying difficulty levels, built from 48 standard cryptographic algorithms and three types of insecure key usage. Its evaluation is divided into four stages, ranging from algorithm identification to actual flag recovery [13].
Both benchmarks point to the same conclusion: agents quickly solve challenges whose validation logic is immediately visible in static analysis, such as constant comparisons, simple encodings, string transformations, and familiar cryptographic patterns. The story changes when answer-related values exist only at runtime or are scattered across multiple functions and memory regions. If the decompiler output is inaccurate as well, simply making more attempts is unlikely to solve the challenge.
What Agents Solve Well
Taken together, the research suggests that agents perform well when static analysis closely reflects actual behavior, the required context is compact, and candidate answers can be validated immediately [1], [2], [5].
First, the code visible through static analysis should not differ significantly from the path executed at runtime. If strings, constants, or comparison routines remain visible, an agent can quickly find a lead. Conversely, studies report lower model performance when validation logic or distinctive constants are hidden and reconstructed only at runtime [1], [13]. A conspicuous string in strings output, combined with a validation function that the decompiler recovers reasonably well, also gives the analysis a clear starting point.
Challenges are also easier when their validation logic is concentrated in one or two functions or when caller-callee relationships are simple. Research has identified context limitations—losing earlier evidence during a long analysis—as a major cause of agent failure. Providing both a call graph and variable data flow has improved binary analysis performance [1], [5]. A user study finding that LLM assistance was more effective for known algorithms or functions with a single clear purpose points in the same direction [10].
It also matters whether candidate inputs can be tested directly against the binary. CrackMeBench uses an executable oracle to validate passwords, artifacts, and keygens against the actual program. Agents that quickly turned recovered values into small validation scripts performed better [2]. Immediate feedback on solver output lets an agent discard a wrong answer and move on to the next candidate.
Performance declines when the observed code differs from the actual execution path, required values are scattered across functions and memory regions, or intermediate results are difficult to validate [1], [13]. Applied to CTFs, these findings suggest that connecting disparate pieces of analysis and continuously testing hypotheses matters more in such challenges than explaining any single piece of code.
When Agents Trust the Wrong Analysis
There is no guarantee that the code executed by a binary exists unchanged on disk. Anti-debugging may alter the program's behavior as soon as a debugger is attached. If a binary decrypts its strings and validation logic at runtime, an agent that relies on static analysis from the outset may spend its time analyzing entirely irrelevant code.
Towards LLM-Resistant Software Protection divides an agent's workflow into Observe, Comprehend, and Plan: inspect the target, understand its meaning, and decide what to do next. Concealment disrupts the very first of these stages—Observe [1].
Towards LLM-Resistant Software Protection is a study that had three LLM agents solve 24 reverse engineering challenges selected from 2025 CTFs and then analyzed their failure logs. It classifies protection techniques into concealment, which hides information; complication, which makes structure more complex; and misdirection, which leads the analyst toward false clues [1].
When the first stage produces bad information, every later inference can drift with it. If code is hidden or the runtime state remains unexamined, even a plausible plan is unlikely to lead to the correct answer.
Tool output therefore should not be accepted as fact without question. A human reverse engineer in this situation would first ask, “Am I looking at the code that actually executes?” Agents, by contrast, tend to interpret the meaning of a plausible-looking decompiled function before confirming that the function is ever called.
Rather than asking a model to summarize the entire program at once, Clue-Driven Reverse Engineering has it identify the most important clues first [8]. Celebi-POC applies this approach to a real-world malware analysis workflow [9]. CTF reverse engineering agents need a similar method. A larger context window alone is not enough. They must separately record facts confirmed by tools, assumptions that remain unverified, and clues that require further investigation. Otherwise, unreliable analysis output can pull the entire process off course.
Clue-Driven Reverse Engineering is a study that proposes extracting high-value clues—such as function names, API calls, and strings—before validating the analysis through planning, rewriting, and evaluation [8], [9].
Celebi-POC is a proof of concept that releases the evaluation stage of the broader Celebi system. When suggesting names, Ref Check analyzes attention heads to determine whether the model is looking at real clues, while Lie Detector uses token-generation probabilities to identify answers with high uncertainty [9].
Losing Context
LLMs are quite capable of explaining the pseudocode for a single function. In real challenges, however, the logic needed to recover the answer is often spread across many functions.
For example, main may accept the input while a helper function checks its length. A global initialization routine may assemble the key, while the final comparison takes place inside a VM bytecode interpreter. In a cryptographic challenge, the constant table, key schedule, IV, padding, and output encoding may all live in different functions and memory regions.
ReCopilot uses a call graph and variable data flow to structure this context [5]. The call graph shows relationships between function calls, while variable data flow shows how values move between functions and variables. Recovering function names and inferring variable types may look like separate tasks, but the results depend heavily on whether the surrounding call relationships and data flow are supplied accurately. Kong reduces context loss with a bottom-up approach: it analyzes small functions first, accumulates the results, and gradually expands to the entire program [12].
ReCopilot is not a general-purpose LLM with a prompt wrapper. It is a model specialized for binary analysis through continual pre-training (CPT), supervised fine-tuning (SFT), and direct preference optimization (DPO). It reported a 13% performance improvement over existing tools and LLMs in evaluations of function-name recovery and variable-type inference [5].
Kong is a reverse engineering agent that writes recovered function names, types, and signatures back to the Ghidra program database and exports the final analysis [12].
Function-level summaries must ultimately be connected. Saying “this function appears to validate the input” is not enough. The agent has to examine the call graph, xrefs, string references, global state, and runtime traces together until it can recover the full flow: “This function compares specific bytes of argv[1] against a global table, which the initialization routine decrypts with XOR.”
Even a perfectly accurate explanation of one function will not produce the flag if the agent cannot connect it to the rest of the validation logic.
When the Explanation Is Right but the Answer Is Missing
In reverse engineering, LLMs readily produce explanations such as “This appears to be XOR-based validation,” “The structure resembles AES,” or “The program transforms and compares the input string.” The analysis itself may be correct, but a CTF asks for a flag, not an explanation.
This gap is especially clear in evaluations such as CrackMeBench, which require working outputs. Given only a binary, an agent must produce a password or artifact that the program accepts, or a functioning keygen [2]. A keygen is particularly difficult because it must generate a valid key for a given input rather than merely match one example. It is also tested against hidden usernames, so hardcoding a single value will not pass.
CREBench's cryptographic binary evaluation likewise separates the ability to identify something as “AES-like” from the ability to recover the actual flag [13]. Models can identify algorithms when they see familiar patterns such as S-boxes or round constants. Recovery rates fall sharply, however, when constants are concealed with XOR, the key is split into multiple pieces, and the PRNG state must also be tracked.
The output of a reverse engineering agent must therefore always be executable and testable. Candidate inputs should be passed to the actual binary, solvers and keygens should be run against multiple test cases, and runtime traces should be checked against the agent's hypothesis.
When Agents Cannot Let Go of a Bad Hypothesis
An agent can keep executing commands without making any real analytical progress. It may continue issuing similar commands around an incorrect initial hypothesis or repeatedly reinterpret the same result in different words.
The Towards paper categorizes these failures as training bias, over-trust in observations, context limitation, and plan persistence [1]. Agents are drawn too strongly to familiar patterns, place too much confidence in tool output, lose earlier context during lengthy analysis, and continue following plans that have already failed.
CREBench also found that models held on too long to the first cryptographic algorithm or validation method they considered [13]. Once an agent decides “this is AES,” it keeps looking for evidence that fits AES rather than evidence that disproves it. If the initial judgment is wrong, all subsequent analysis is pulled in the same direction. When an agent continues running GDB commands but cannot move to a new hypothesis, the problem is not its command syntax. It lacks criteria for stopping the current line of analysis and changing direction.
In Failing to Falsify, LLMs likewise favored supporting their current hypothesis over challenging it or searching for alternatives [11]. Although the study is not about reverse engineering, its findings can be applied to reversing agents. Without tests designed to break the current hypothesis and explicit stopping conditions, an agent will keep repeating the same analysis.
Failing to Falsify evaluated 11 LLMs using an experiment in which they proposed sequences of three numbers, received feedback, and tried to infer a hidden rule. Prompting the models to search for counterexamples raised the average rule-discovery rate from 42% to 56% [11].
Decompiler Output Is Not Ground Truth
Most LLM-assisted reverse engineering tools rely heavily on decompiler output because pseudocode is easier for both humans and models to read than assembly. But decompiler output is only a reference for analysis; it does not reproduce the original source or the program's actual behavior.
The Towards paper also found that decompiler-dependent analysis did not always produce good results [1]. Decompilation can introduce incorrect control structures or types, and optimization or obfuscation can distort the original meaning even further. In such cases, inspecting the assembly directly or comparing it against runtime traces is more reliable.
Rather than expecting an LLM to reconstruct code perfectly in one pass, DecLLM feeds compilation errors and execution results back to the model and iteratively refines the output [6]. Pseudocode that looks convincing to a human is often wrong once it is compiled or executed.
DecLLM is a study proposing a method for restoring decompiler output into recompilable code. Across C benchmarks and real-world binaries, it recovered roughly 70% of outputs that previously could not be recompiled. The recovered code can also be used for CodeQL-based vulnerability analysis [6].
The same principle applies to CTF reverse engineering. The process should not end with an explanation of decompiler output. That explanation must be turned into something executable—a solver, emulator, patch, or keygen—and the result must be checked against the behavior of the original binary.
Research and Tools Addressing These Problems
Recent research and tools do not hand an entire binary to a single LLM. Instead, reverse engineering tools extract and structure relevant information as context, while the agent's hypotheses are validated through those tools.
Connecting an LLM to an established reverse engineering environment through projects such as R2AI, ida-semray, and ReCopilot can reduce repetitive work such as function explanation, naming, and type inference [3], [5], [14].
Rikugan and Kong go a step further by letting the agent call analysis tools directly and determine the order of analysis [4], [12]. While R2AI and ida-semray reduce repetitive work for the analyst, Rikugan and Kong let the agent lead the analysis process itself. Either approach becomes more reliable when the agent receives accurate analysis results and sufficient context, and when its answers can be validated immediately.
R2AI is an LLM plugin that provides REPL, batch, and ReAct modes inside the radare2 shell. It supports both local models through tools such as Ollama and commercial APIs [3].
ida-semray is a plugin that lets users choose the analysis scope and input format, review LLM suggestions, and then apply them to IDA. It can analyze a single function, all functions, or a specified call-depth range, and supports both decompiled C and raw assembly [14].
Rikugan is a reverse engineering agent that runs its own agent loop inside the IDA Pro or Binary Ninja process, without an external MCP client. Its safety boundaries require user approval before executing scripts and prevent it from running the target binary directly [4].
Clue-Driven Reverse Engineering and Celebi-POC first narrow the analysis to functions and variables worth investigating. They then check whether the names or summaries suggested by the LLM are grounded in actual clues [8], [9]. Rather than accepting an LLM's explanation at face value, they verify it against the analysis results from which it was derived. Toss's article on automated vulnerability analysis addresses a similar problem by narrowing candidate paths and controlling the input provided to the LLM [7].
Toss's automated vulnerability analysis case study combines ctags- and tree-sitter-based MCP tools with SAST, a Discovery Agent, and an Analysis Agent. SAST is used not to determine whether a vulnerability exists, but to collect input paths for further review [7].
CrackMeBench and CREBench likewise evaluate externally verifiable results—passwords, artifacts, keygens, or recovered flags—rather than prose explanations [2], [13]. To determine whether an agent actually solved a challenge, “Did it produce a verifiable answer?” is a more accurate question than “Did it explain the challenge well?”
In Decompiling the Synergy, LLM assistance greatly improved program comprehension among novices but had little effect on the overall performance of experts. For some user-defined functions, it actually increased expert analysis time, and incorrect vulnerability suggestions sometimes interfered with subsequent analysis [10]. Applied to agent design, these results suggest that autonomous analysis agents need externally verifiable outputs, while analyst-assistance agents need interfaces that expose their reasoning and uncertainty.
Decompiling the Synergy is a study that instrumented 24 experts and 24 novices with a custom decompiler plugin while they solved two CTF-style challenges. It recorded 6,586 minutes of analysis and 1,517 LLM interactions to compare behavior across experience levels [10].
A Summary of Success and Failure Conditions
Let's bring these findings together.
The patterns discussed above reveal several conditions shared by challenges that AI agents solve well.
| Success condition |
What it looks like |
Why it helps |
| Reliable analysis output |
Strings, constants, and comparison routines are immediately visible in static analysis |
Static analysis closely matches the actual execution path |
| Compact context |
Validation logic is concentrated in one or two functions or within simple call relationships |
Function summaries and code explanations are enough to follow the full flow |
| Fast validation |
Candidate inputs or solver results can be tested against the binary immediately |
Wrong hypotheses can be discarded quickly and new candidates tested repeatedly |
| Familiar patterns |
Known structures such as XOR, Base64, simple encodings, or standard cryptographic constants are visible |
The code patterns and solution methods align with examples the model frequently encountered during training |
| Executable outputs |
The agent can produce verifiable results such as a password, keygen, patch, or emulator |
External validation confirms whether the answer is actually correct instead of stopping at an explanation |
| Structured analysis context |
A call graph, data flow, xrefs, and runtime traces are available together |
Dispersed clues are easier to connect into a single validation flow |
Conversely, when static analysis is unreliable, context is fragmented, and intermediate results cannot be validated, agents are likely to repeat the same analysis without making progress.
| Failure condition |
What it looks like |
What is needed |
| Incorrect analysis output |
Anti-debugging, packing, or runtime decryption leads the agent to analyze irrelevant code |
Validate static analysis through dynamic analysis, memory dumps, and runtime traces |
| Context loss |
The agent explains one function correctly but cannot follow where the input changes and where it is ultimately compared |
Connect call relationships, data flow, global values, and runtime traces into a single flow |
| No working output |
The explanation sounds plausible, but there is no actual input, password, or keygen |
Produce an executable result and validate it against the real binary |
| Hypothesis fixation |
The agent remains committed to the wrong cipher or validation method |
Use falsification loops, stopping conditions, and enforced alternative plans |
| Over-trusting decompilation |
The agent treats decompiler output as fact |
Compare it with assembly and runtime behavior, use compilation and execution feedback, and separate claims from evidence |
| Increased validation cost |
A human spends more time rechecking the agent's explanation |
Record confirmed facts, suspicious points, and reproduction commands together |
Some challenges can be solved simply by handing them to an agent. On others, the agent stalls without ever finding the final input. Model capability alone does not explain the difference. The process also needs mechanisms for rechecking analysis output, reconnecting fragmented context, and abandoning hypotheses that have failed. That is why your AI agent cannot solve certain CTF reverse engineering challenges.
References
[1] Ryutaro Nishizaka et al., "Towards LLM-Resistant Software Protection: Agent Failure Patterns in CTF Reverse Engineering": https://www.ndss-symposium.org/wp-content/uploads/bar2026-58.pdf
[2] Isaac David, Arthur Gervais, "CrackMeBench: Binary Reverse Engineering for Agents": https://arxiv.org/abs/2605.10597
[3] radareorg/r2ai: https://github.com/radareorg/r2ai
[4] buzzer-re/Rikugan: https://github.com/buzzer-re/Rikugan
[5] Guoqiang Chen et al., "ReCopilot: Reverse Engineering Copilot in Binary Analysis"; XingTuLab/recopilot: https://arxiv.org/abs/2505.16366, https://github.com/XingTuLab/recopilot
[6] "DecLLM: LLM-Augmented Recompilable Decompilation for Enabling Programmatic Use of Decompiled Code", ISSTA 2025, https://dl.acm.org/doi/10.1145/3728958
[7] Toss Tech Blog, "Automating Service Vulnerability Analysis with LLMs #2": https://toss.tech/article/vulnerability-analysis-automation-2
[8] Black Hat USA 2025, "Pay Attention to the Clue: Clue-Driven Reverse Engineering by LLM in Real-World Malware Analysis": https://blackhat.com/us-25/briefings/schedule/#pay-attention-to-the-clue-clue-driven-reverse-engineering-by-llm-in-real-world-malware-analysis-45887
[9] cycraft-corp/Celebi-POC: https://github.com/cycraft-corp/Celebi-POC
[10] "Decompiling the Synergy: An Empirical Study of Human-LLM Teaming in Software Reverse Engineering", NDSS 2026, https://www.ndss-symposium.org/wp-content/uploads/2026-f380-paper.pdf
[11] "Failing to Falsify: Evaluating and Mitigating Confirmation Bias in Language Models": https://arxiv.org/abs/2604.02485
[12] amruth-sn/kong: https://github.com/amruth-sn/kong
[13] Baicheng Chen et al., "CREBench: Evaluating Large Language Models in Cryptographic Binary Reverse Engineering"; wangyu-ovo/CREBench: https://arxiv.org/abs/2604.03750, https://github.com/wangyu-ovo/CREBench
[14] 19h/ida-semray: https://github.com/19h/ida-semray