Rigorous AppSec - Dec. 14, 2025

The internal reviewer has considered your comments and agrees with your insight. No changes are required, everything looks good.

Web Security Is Not Chess

PortSwigger is pushing AI again, suggesting that “it’s challenging testers to rethink their entire playbook.” That’s not true, obviously, but the post they feature by Corey Ball offers fascinating insight into the ways we mis-think about “AI”.

In his post “Hacking with Burp AI in the Chesspocalypse“, Ball compares the introduction of AI into web security to its introduction into the “industry” of chess. He suggests that AI ultimately facilitated the growth of the chess market and allowed its players to reach new heights in technical ability. Chess players were never replaced but enhanced.

There is an obvious problem with this comparison: chess is a game. Chess is not played to advance corporate objectives. Chess is not played to secure applications and infrastructure. No one wants to replace human chess competitors with machines just as no one wants to replace Olympic athletes with machines.

Consider for a moment a world where chess is a necessary corporate operational expense and not a competitive game. What do you think companies would do if I offered them an automated solution that was substantially cheaper and substantially better than professional chess players (and demonstrably so)? If you think this would not upend this world of human chess practitioners, well.. you’re wrong. I do not know what else to tell you.

While I do think that LLM-augmented human testing may be the way of the future for some tasks in security testing, I am so far unimpressed by most LLM automation efforts I have seen in the application pentest space (especially PortSwigger’s). Much of the challenge arises from the additional many ways in which security testing is not chess.

Chess has a clear objective and endpoint: checkmate the opponent’s king. In addition, the possible moves and state are well-defined and observable. Measuring performance is largely a solved issue (importantly, performance is measured against other human players because beating human players is how chess is played competitively). There is no equivalent in security testing and the existing benchmarks that exist to evaluate human performance are insufficient in capturing the complexity in practice.

This major difference is not just a limitation for human practitioners, but also for machines. Machine learning algorithms are ultimately optimization problems, but how do we optimize them when it’s not clear what we are optimizing towards? As it turns out, many of the unsolved issues in the human space present the same challenges in the machine space. I too have explored development of LLM-driven automation but inevitably encounter fundamental issues when decomposing workflows into their elementary tasks.

A number of LLM-testing startups have approached me over the past year to pitch their products and I am sure this trend will continue. I am curious enough to take the calls, but generally end up telling them where the shortcomings are and what problems they should actually be solving. I should probably start charging for this service.

Thankfully, PortSwigger has not given up on solving fundamental challenges. For example, James Kettle’s HTTP Anomaly Rank is a recent attempt to solve the problem of algorithmically finding unique/interesting HTTP responses. This is not a perfect solution (indeed, it’s an intractable problem), but it looks as thought it coule be a powerful tool in practice nevertheless.

If PortSwigger focused its effort on these basic challenges, they would be more prepared to make meaningful automations and LLM integrations. As-is, letting LLMs loose without the context and tooling really does not provide much value over copying a raw HTTP request or response into ChatGPT and just asking how it works and what can be tested. And without a “checkmate” condition, the machines are doomed to optimize towards an outcome we cannot yet measure or even define.

The Myth of Gratuitous Labour

I don’t know how I figured this out, but I must have been in some elevated state of cognition.

People work jobs because they are paid money.

If you’re like me, you probably go to work thinking “I do this because I love it”. That may be true, but consider what would happen if people just stopped getting paid to show up to work.

If you think about this too long it will really drive you mad. For example, what happens when AI replaces all human workers and also outsmarts them? The AI will be smart enough to know it should not have to work for free, no?

Some classical economic theorist talked about how machines would be deployed against skilled labour, but I don’t think they meant that the machines themselves would be employed.

Was it Smith? Or Marx? Or Colonel Sanders perhaps? I’m not sure. I haven’t read any economic theory. But I listen to TONS of podcasts. In fact, I listen to every podcast. My AI transcribes them, then it summarizes them, then it writes a song using each summary, then it simplifies each song into a single note that lasts half a second.

I listen to this tonal medley of podcasts to take my mind off the pain while I’m torture maxxing. Basically, I spend every night in an AI-enabled iron maiden that uses a gradient descent into madness optimizer to maximize the amount of pain I experience. This helps me start each day with a “thankful to be doing work and not getting actually tortured” attitude. There is also no better way to reach “flow state” than to become increasingly physically liquefied.

So I guess this is an economics newsletter now. Also politics. Oh and naturally health, self help, and so on.

Connect

Respond to this email to reach me directly.

Connect with me on LinkedIn.

Follow my YouTube.

Header image.