Is a Rabbit Hole Difficulty, or Just Frustration?

Introduction

This post is a written version of my talk, "Is a Rabbit Hole Difficulty, or Just Frustration?", given at the CCA Korea 2026 National Liberation Day seminar in August 2026. To start with the conclusion: a rabbit hole is neither good nor bad in itself. I think it is difficulty if you can disprove it, learning if it makes sense afterward, and frustration if there is no way back.

It started with a question that came up while solving a challenge: "Is this path really part of the solution, or is it a fake road built to burn my time?" In recent CTFs, I kept digging into suspicious features or complex attack surfaces that turned out to have nothing to do with the actual solution. After this happened again and again, I wanted to think the question through properly.

In my case, it was a web challenge with a memo page and a bot. A memo page plus a bot, so of course I assumed it was an XSS challenge, and I tried all kinds of chaining and tricks to make it work. I spent the entire competition on XSS and on every attack I could think of against that memo page. In the end, the memo page had nothing to do with the solution; the challenge was actually a CSS leak problem. Looking back, the familiar pattern "memo + bot = XSS" dominated my first impression, and every time I got stuck I changed my attack method instead of my hypothesis.

I have run into the same phenomenon from four positions. As a player, I solved challenges. As a challenge author, I wrote and ran challenges for several competitions, including YISF (Youth Information Security Festival). I have also researched automating CTF solving with AI agents, and while running club training sessions I watched learners get lost. The same "wrong path" looked completely different depending on where I stood, and this post is a record of those four viewpoints. I want to make clear up front that this is not a post that hands down the right answer; it is my current thinking, based on my own experience and the opinions of people around me.

What is a rabbit hole?

In this post, a rabbit hole means "a state where you sink deep into a plausible-looking hypothesis or path, stop seeing other possibilities, and keep spending time on it." It is often understood only as a time-wasting trap planted by the author, but that definition is too narrow. Ambiguous server responses, tool false positives, differences between local and remote environments, and a player clinging to a single hypothesis all create rabbit holes just as well.

Here is how I will distinguish three terms that are often used interchangeably. These are not precise academic definitions, just distinctions for use in this post.

Term	Meaning in this post
Red Herring	A fake clue placed to draw attention elsewhere
Decoy	A fake path or attack surface deliberately built by the author
Rabbit Hole	The phenomenon of a player actually sinking deep into a particular hypothesis

The key point is that a decoy can create a rabbit hole, but not every rabbit hole is a decoy. If a player can rule out a fake clue right away, it is not a rabbit hole. Conversely, even if the author never intended it, if a player stays stuck in it for a long time, it is a rabbit hole.

How you fall in: "Just a little more and I'll have it"

Falling into a rabbit hole usually follows the same loop.

You find a suspicious clue.
You form a promising hypothesis from it.
You repeat analysis and exploitation.
You read failure as "the hypothesis is right, the payload is wrong."
You build a more complex attempt and go back to step 3.

The problem is not failure itself. The moment you see a failure and read it as "I'm almost there" rather than "my hypothesis is wrong," it becomes hard to climb back out. People and LLMs run this loop in almost exactly the same way, which connects to the AI perspective later in this post.

Why ask this now?

When I first started writing challenges in 2023–2024, the standard I learned for a "good CTF challenge" was clear: reduce unnecessary guessing, clean up unintended elements, and let players reach the solution through skill and ideas. If I included functions or APIs that were never used in the solution, or an unexplained fake flag, reviewers asked me to "remove it if it isn't actually used in the solution."

Then LLMs entered CTFs. That does not mean every challenge solves itself, but for a challenge with a familiar structure and enough source code, an LLM can drastically cut the time from analysis to writing the exploit. Clean, honest challenges started becoming easy for LLMs too.

Perhaps because of this, in the competitions I have played recently, "fake roads" have started to show up again. People talk about deliberately adding vulnerable-looking unused features, attack surfaces unrelated to the answer, and paths that make LLMs waste their effort, as a way to block LLM solving (Anti-LLM). I am not claiming every competition does this. But the starting point of this post is that decoys do not only get in the way of LLMs; they get in the way of people too.

One rabbit hole, four experiences

Player: in the end, it's about time

For a player, I think a rabbit hole is less a question of skill than of where to spend time. Competition time and team members are limited, and spending 30 minutes on one hypothesis means losing 30 minutes you could have spent on another. And the longer you spend, the harder it becomes to drop the hypothesis: "I've come this far," "maybe only the payload is wrong."

At this point, a player is not just digging down a suspicious path; they keep "discovering" evidence that looks suspicious. They keep getting pulled along by the first trace they saw, mistake tiny changes in a response for progress, and sometimes several teammates end up re-verifying the same hypothesis.

That said, not all digging is pointless. If you learned why a wrong path was wrong, you come away with a sense for telling similar vulnerabilities apart and a standard for what to look at first next time. So what a player needs is less the ability to avoid rabbit holes and more the habit of noticing quickly and coming back. I summarize that habit later under "Questions for getting out."

Author: hard is not the same as unfriendly

For an author, I think a rabbit hole is a tool for designing difficulty. It keeps the solution path from being exposed right away, makes players think about what to look at first, and blocks solutions that just plug in a single known technique. The people who add them have clear reasons for doing so.

But a hard challenge and an unfriendly challenge are different things.

Hard challenge	Unfriendly challenge
Requires deep knowledge	Offers no basis for telling things apart
Has many reasoning steps	Requires information only the author has
Requires verification	Is easily confused with an infrastructure error
Leaves clues even after failure	Still makes no sense after solving it

What authors usually look at in review seems to be the intended solution and unintended solutions. But what shapes the player experience the most is a third thing: "the path that isn't the solution but looks far too plausible." That is why a review should also ask, "How many minutes will players spend on this fake path?"

Another dividing line is whether a fake feature was "added" or whether it "arises" from the real structure. A rabbit hole that naturally comes from a feature the service actually needs makes far more sense than meaningless code or an unrelated vulnerability forced into the challenge. In the end, I think the question an author should ask is whether the device tests the player's analytical skill, or just their ability to read the author's mind and their patience.

AI: being wrong more convincingly

LLMs produce a plausible first hypothesis very quickly. That is a strength, but it gets expensive when the hypothesis is wrong. LLMs often interpret tool output to fit their first hypothesis instead of trying to disprove it, and when they fail, they blame filters, encoding, or the environment and build ever more complex payloads. They run the same loop described earlier, only faster and in more convincing sentences.

So if you just tell an AI "solve this," you may get confidence without evidence instead of an answer. You need to also ask "What if this hypothesis is wrong?", "What is the simplest experiment to show this is fake?", and "When do we drop this hypothesis?", and have it write down what it actually verified separately from what it is guessing. At the same time, rabbit holes block mindless automated solving and once again call for a human in the middle to review candidates and cut them down. This deserves its own discussion from an AI agent research perspective, so I only touch on it briefly here.

Education: when does failure become learning?

From an education perspective, the goal is not to keep learners from getting lost. It is to make sure the time they spent lost turns into judgment.

Productive failure	Pointless digging
You can explain why it was wrong	You can't tell why it was wrong
Leaves a standard for the next challenge	Leaves only repetitive work
Helps you understand the differences between similar techniques	The only lesson is "reading the author's mind"
Makes sense after solving	Still makes no sense even after seeing the answer

Hints can be judged by the same standard. "Use feature A" gives a direction, while "If your current hypothesis is right, what difference should you see in the response?" gives a way to verify. I think a good educational hint tells you where you can stop looking, rather than where to look harder.

I asked six people

To avoid drawing conclusions from my own view alone, I asked six people who work or study in security the same question: "In CTFs, is a rabbit hole a necessary form of difficulty, or a device that lowers the quality of a challenge?" This was not a formal survey; these are individual opinions summarized anonymously, and they do not represent the security community as a whole.

The opinions fell into roughly three camps.

Position	Responses	Main reasoning
Necessary, or realistically unavoidable	3	Blocks mindless LLM solving and creates a human-in-the-loop who reviews candidates
Acceptable under conditions	2	Must be shallow and verifiable, and fit naturally with real features
Lowers challenge quality	1	Designing around AI undermines the essence of a human skill competition

Those in favor emphasized that a rabbit hole is not just an obstacle; like a real environment, it makes players check multiple attack points themselves. But the same caveat came up repeatedly across answers: a person should be able to tell it is fake at a reasonable cost.

The strongest objection cannot be brushed aside either. It argues that the moment challenges are built around Anti-LLM, a CTF can turn into "a competition of designing paths for AI to waste time on." You can build challenges that humans solve better without rabbit holes, but that takes far more resources to design and review, and if a decoy hides the core idea of a challenge, the tail is wagging the dog.

Positions differed, but some common conditions came up again and again.

A fake path must not be deeper than the real flag path.
There must be a clue that lets you be sure it is a rabbit hole.
Checking that clue must be cheap.
It must fit naturally with real features.
Verifying it should leave you with analytical experience.

The first one in particular, "a rabbit hole should be shallower than the real solution," seemed to be the most intuitive standard.

Good rabbit holes and bad rabbit holes

My conclusion is that we should not judge rabbit holes by whether they exist, but by "Can you come back? Does it make sense?" The standards below are not objective laws; they are my current judgment, combining my experience with the six opinions.

Good rabbit hole	Bad rabbit hole
Shallower than the real solution	The fake path is more elaborate than the real solution
A simple experiment can show it is fake	Indistinguishable from an infrastructure error
Exists naturally in the service structure	Forced in purely to obstruct AI
Other clues lead you back	Getting out requires information only the author has
Leaves a standard for judgment even if you fail	Only burns time and tokens
Makes sense after solving	Even the writeup doesn't explain why

Questions for getting out

Before testing a hypothesis, decide on a failure condition as well as a success condition. An attempt with no decided point at which to give up is closer to a prayer than an experiment.

If this hypothesis is right, what should I observe?
What result would make me drop this hypothesis?
Is the result I just got evidence, or an illusion?
Is the failure a problem with my implementation, or with the hypothesis itself?

A good player is not just someone who attacks well, but someone who can quickly fold a wrong hypothesis and come back. When you hand work to an AI, you can pass these same questions along as part of the prompt and stopping conditions.

A checklist for authors

Before adding a rabbit hole, I suggest checking the following six things. If even one is completely missing, that rabbit hole is likely to feel more like frustration than difficulty.

Depth: Is the fake path no more complex than the real solution?
Fake check: Is there a minimal experiment that shows it is fake?
Way back: Can other clues bring players back to the solution path?
Cost: Is the time it demands from people reasonable for the difficulty?
Naturalness: Does it fit the challenge's features and setting?
Makes sense: Would a player who reads the solution agree, "Ah, so that's why it wasn't it"?

Order matters too: ask "What does this demand from people?" before asking "Does this block LLMs?"

There is no answer yet

I think CTF challenge design is currently facing a problem with no easy solution. A clean structure is fair to people, but LLMs understand it quickly too; a complex decoy raises the cost for LLMs, but it also makes the experience worse for people. There is no way yet to get both at once, and rabbit holes may be a stopgap that emerged in between. I think this is not a problem caused by lazy or malicious authors, but a hard problem that came from a changing environment.

So the balance I have in mind is not to blindly hide the right path, but to make wrong paths verifiable and recoverable. A challenge with overly obvious clues has no tension, and a challenge full of traps is just unpleasant. The goal is not "make it unsolvable even for AI," but to leave a verification path that people can accept.

I also see a bigger shift coming. Future CTFs may evaluate not only finding vulnerabilities and writing exploits, but also choosing which AI-generated candidates to trust and stopping wrong hypotheses. Whether that is an expansion of CTF skill or the start of an entirely different game, I don't know yet.

To sum up: a rabbit hole is difficulty if you can disprove it, learning if it makes sense afterward, and frustration if there is no way back. Since this is a question without a single right answer, I gathered four perspectives and several people's opinions to organize my own thinking. How do you see this question? I would love to keep this conversation going with more people.