There is a classic Simpson’s bit where the inept physician (Dr. Nick) advertises a flat fee for any surgery. The advertisement exposes the shoddy nature of Dr. Nick’s work through a brain surgery patient struggling with a Chinese finger trap. The trap works because escape is unintuitive; pulling your fingers further apart only tightens the grip of the trap. The audience can recognize that failure to overcome and escape is – to some degree – a cognitive failure. As there are only so many movements you can make in the trap, it should take little time to exhaust the possibilities and escape even if you don’t understand the underlying mechanism.

The novelty and relative popularity of Chinese finger traps (compared to all other types of finger trap) is owed to the initial unintuitive nature, but also the tight bounding of the problem space that lends well to a constructivist-oriented learning approach. With minimal guidance, you can (hopefully) discovery for yourself how to successfully operate and ultimately escape the trap. The fun is in watching the struggle which eventually gives way to the learning process. But is this an effective way to learn?

You could run an experiment here to determine what is the most effective approach to achieve a specific learning objective. What is the objective? To teach novices how to escape a Chinese finger trap. In this case, the most effective approach is the one that achieves this objective in the shortest time. To make the experiment more likely to produce a strong signal, it would be wise to choose a target population where there is a more prominent learning curve to be expected. I would recommend choosing young children here to avoid the potential for your scientific decisions to be interpreted in any way as offensive. Science is not a joke.

In this case, we will run only a thought experiment. In group A, you have children learning independently. You apply the trap, start the timer, and watch them struggle. The timer stops when they escape. In group B, you apply the trap, start the timer, and then teach them how to escape. Similarly, the timer stops when group B children escape. Which group is likely to be more efficient? If you think the uninstructed group will be more efficient, I have some Polymarket orders to sell you.

Advocates of the discovery-based learning method may protest here. Perhaps they would argue that there is additional value in the learning process that is not captured by my crude outcome (who escapes faster). Could our children better learn how to learn independently? Fair point, but quite frankly, I don’t care. I run a children-escaping-from-chinese-finger-traps factory. The lives and future of those children are in my hands, but only to the point where they escape finger traps. If the instructional method is more effective, that’s what I am going to do. After all, teachers are not exactly a significant expense these days.

I consider finger traps to be a steel man in making this point. The problem space is bounded and the traps themselves offer immediate tactile feedback as to whether they are tightening or loosening (effective and immediate feedback is generally recognized as a component of effective practice). Let’s now consider Capture The Flag (CTF) style exercises as learning tools to train security testers. What are CTFs and are they good?

Traditionally, CTFs originate from competitive hacking events (DEF CON), where competitors operate within a purpose-built system or networked environment and attempt to navigate the technology in such a way to uncover hidden information (in the form of a flag). Today, CTFs take many different forms. When I refer to CTFs or CTF-style exercises throughout, I am considering exercises where the participant must execute one or more security testing techniques in order to achieve a single outcome such as retrieval of a flag. There are many different competitive and learning CTF formats that I will not fully explore. I will make the distinction between mystery CTFs where the skill/technique to be applied is unknown and educational CTFs where the skill/technique is known and the exercise itself is prefaced with relevant instructional material (naturally, there is a spectrum between). A simulated environment is one where multiple CTFs are embedded across a network or application in a way to mimic a realistic environment.

Because I am no longer in the finger trap business and now manage a security testing factory of sorts, I am interested in the question of whether CTFs are effective learning tools for security professionals. If, like me, you are from the Wikipedia generation, you might take their word that “CTFs have been shown to be an effective way to improve cybersecurity education” and call it a day. But if you are also a real science sicko like me, this just does not satisfy.

Wikipedia primarily cites 3 studies in support of the claim that CTFs are effective learning exercises. The first we will look at is probably the strongest contender. Even so, this is one of the poorest methodological write-ups I have ever seen and even if we ignored the major methodological flaws (I may write in-depth on this another time), the results do not support the claim. We wouldn’t be on this journey otherwise, would we?

The study employed a single group pretest-posttest design, recruiting a small (n = 10) sample of undergraduate students. Besides being a small sample, the percentages reported do not align with n= 10, so the authors either made mistakes or committed some minor scientific fraud (the issue I am about to highlight is actually an effective technique to identify fraud). For example, 47.6% are reported as junior-level. How exactly do we divide 10 people into a group representing 47.6%? Even if we assume they are providing data on the class overall, they report 24 total students and 11/24 rounds up to 46%. This alone is sufficient in the academic world to warrant at least a correction, but the process to force a correction is even worse than reporting vulnerabilities to Microsoft.

The authors deployed a simulated environment (network) with a variety of basic security issues. They asked four questions (hypotheses), which was four more than the methodology could handle:

None of the study tools (besides samples) were published, so we know very little of the actual questions and how they were posed to participants (a pretty big failure). We can determine that students were asked about time, but we can only infer that they were likely first asked how much time they anticipated spending and later asked how much time they actually spent. The median participation time reported was 20 hours. More on this soon.

The strongest result in support of the conclusion comes from the assessment of “practical” skills. How did they assess practical skills? Students were asked to explain terms and techniques with long-written answers. Ignoring that there was no indication of blinded grading, evaluating someone’s ability to explain “port scanning” (yes, this was one of the questions) is hardly what I would consider an effective assessment of practical skills. There is a bigger issue here, however.

Recall that the median effort was 20 hours. That is not a small amount of time. What were students doing? Were they running nmap the entire time? Were they reading man pages? Were they watching YouTube? Were they misreporting time because the study is part of their course work and they wanted to give the impression that they are working hard? We have no idea. That’s a problem, obviously.

Even if we assume that students did improve their knowledge of port scanning, we cannot attribute it to a specific activity. Trainees typically do not learn how to use nmap by running it blindly with no prior knowledge until they figure it out. They learn by reading (or watching) existing training material or other documentation. Does time spent learning how to do something in order to solve a CTF count as learning from the CTF? Ridiculous!

Ultimately, this study tells us little. It certainly does not demonstrate that CTF-based learning is effective. To prove that I am not imposing an impossible standard, I will even follow an important scientific criticism principal and detail what I believe would be a more robust experimental design.

This study could have organized two groups. The first group would participate as originally designed, but the second group would not participate in the CTF exercise. Instead, they would be assigned relevant readings (including what port scanning is and how to do it) and would be given the opportunity to practice the skills against live systems (non-CTF). Both groups would be required to spend the same amount of time, which is important to control when comparing effectiveness. Personally, I would rather track time spent in a controlled environment than depend on self-reports.

This would be a much more robust design because we better isolate what we are interested in observing an effect from (learning via CTF), but we didn’t get that. Instead, we got scienceslop. We got a mess of a paper and experiment that was nevertheless sufficient to be cited by Wikipedia as solid evidence that we have learned something about the world even though we’ve learned nothing. But wait, aren’t there more supporting studies?

For study number 2 cited on Wikipedia, we have an experiment on high school students. No wait. There was no experiment and no results. OK. How about study 3? Once again, there is no empirical contribution, no methodology, no results.

Looking beyond Wikipedia, what about this 2022 study titled “Effectiveness of CTF education”? Sounds perfect. The authors recognized the lack of measurement of learning outcomes from CTF-based exercises and developed a framework and pilot study to investigate. To their credit, they developed an interesting approach to capture information contextual to the learning process and participant knowledge and attempted to gather data across two multi-day CTF competition events. To the authors’ credit (and unlike almost ever other paper in this space), they were forthright in admitting that the data they were able to collect was insufficient to support any particular hypothesis. Their conclusion of skill development over the course of a CTF? “No evidence for this has been observed.

We could go on, but what would be the point? There appears to be broad acceptance that CTF-style exercises have educational or skill development value, but no one has been able to robustly demonstrate this. Importantly, there is substantial divergence in the study populations, the structure and style of CTF exercises, and even the intended learning outcomes and objectives. No study has seemingly ever examined professional security testers (or trainees) and no study has ever utilized objective measures of technical skill. Do we simply need a competent researcher to put together the final, definitive study with a robust methodology. No. I don’t believe we do.

While cybersecurity researchers have been testing the waters of scientific research, researchers in more mature disciplines have furthered the foundations of human knowledge in education and technical skills training. We know that heavily guided instruction from experts is more efficient in developing complex skills and knowledge. We know that this effect recedes with the development of sufficient expertise. We know that effective practice should be structured towards gaps and provide effective feedback. Do CTF exercises satisfy any of these? Not really.

I will say that instructional CTF exercises, such as those offered by PortSwigger’s Web Academy, are a likely to be much more valuable; the labs are preceded by detailed instruction (probably where most time is spent), labs are bounded to a specific test case (or issue), and some feedback is available (in the form of a solution). This is not an ideal learning environment, but it is far preferred to environments where trainees waste precious learning time simply because they don’t know what they don’t know. Perhaps CTF is not even the proper term to describe what is effectively just practice in a lab environment.

What is the ideal learning environment? Well, I’m not so sure; we haven’t done the science. That said, I have finally adapted (stolen) an approach used by surgeons to train and evaluate their novices. I just presented the proof-of-concept and pilot study at OWASP Global AppSec Vienna, and am now looking to run a more rigorous experiment to better evaluate the efficacy. Do you manage or train security testers? Do you know someone who does? Please connect with me and we can advance the state of the art.

Respond to this email to reach me directly.

Connect with me on LinkedIn.

Follow my YouTube.

RSS feed here.