The gap between AI governance and hands-on reality
- General
- by Jacob Riggs
- 29-09-2026
This year I took on new certifications in effort to upskill myself in AI security, something I've become heavily invested in and firmly believe will likely define the next decade of this industry (and a good deal of what comes after it).
I came away with two certs - the Advanced in AI Security Management (AAISM) from ISACA and the Certified Offensive AI Expert (COAE) from HackTheBox:
These were chosen deliberately. Like most of my past research, I wanted to treat AI as another discipline and study it properly. This meant covering the governance and management side (where the policy, risk, and assurance questions seem to converge), and conversely the hands-on offsec side, where those same decisions meet reality and either work or fall apart.
In short, the AAISM shows how organisations are currently being told to defend against AI, and the COAE helped me understand how quickly that advice can fail when tested. That gap between the governance and hands-on reality is what this post is really about.
Enter the governance side
The AAISM sits somewhere on a spectrum (much like myself), and deals with many of the management related questions that make security leaders uncomfortable. Like how you frame AI risk, how you fold it into an existing security/assurance program, how you govern the models and the data that feeds them, and how you answer to a regulatory picture (the EU AI Act and the others that will follow it etc). It's sober material, and I'd recommend it to anyone whose job now involves signing off approvals for AI systems they didn't build or might not fully understand.
In principle this type of governance is great, but in practice it's basically a map drawn of an area that keeps rapidly changing. I've found it commonly describes a posture that assumes a level of control very few orgs actually have, as useful frameworks arrive slower than the technology/people/processes they're intended to cover. So by the time a control has been written up, socialised, and adopted, the capability it was meant to constrain has already changed shape.
That doesn't make all governance worthless. It's still entirely necessary, and someone has to hold the line on accountability, but it describes where the fight ought to happen rather than what actually happens once one starts.
The offensive side
The COAE is a heavy hands-on offsec AI exam covering direct/indirect prompt injection, coercing models into leaking their own instructions, turning a model's output back against the application that trusts it, abusing the tools and agents we're now wiring these models into, and poisoning the data and pipelines behind them. I spent a large part of my free time this year immersed in studying these vectors (and of course trying to navigate all the new fancy buzzwords).
In the end I passed, but this one humbled me. After the best part of 15 years of hacking into things, I expected my existing hands-on knowledge would carry me through, but it didn't. Many modules genuinely forced me to properly understand how these systems fail rather than just pattern-matching attacks from the web and infra work I already knew. I finished with a lot more insight and respect for the technical depth involved here than I started with.
I think the thing that stayed with me most, given I've now spent enough time doing this, is the uncomfortable asymmetry.
An attacker needs a single input that works. The defender, however, has to anticipate an almost infinite space of them, expressed in ordinary language, against a model whose behaviour nobody can fully specify (often not even the people who trained it). Traditional appsec was hard, but it was at least deterministic and mostly bounded (you could actually enumerate the inputs that mattered). The syntax an injection had to obey was fixed by a known grammar, and the parser reading it was therefore deterministic. There is no equivalent for an LLM. The attack surface has become anything a model can be talked into, and that is not a surface you can enumerate.
The guardrails most organisations lean on only seem to make this worse, because the majority of them are best effort prose, rather than actual controls. I've noticed the boundary often relied on was the wording of an AI generated system prompt, or in a courteous instruction to keep something secret, or in the assumption a tool wouldn't echo its output.
A control you can talk your way past is not an effective control.
Why I think defense is losing the race
My controversial hot take, with an absence of humility and some natural bias (coming from an offsec background), is that AI governance is sliding into glorified compliance theatre. That's not aimed at the discipline or the people who do it well. My issue is with the abundance of box-ticking from self-proclaimed SMEs. A frustration the best security leaders tend to share, and lends itself to a wider problem - that we all (regulators included), need to pay closer attention to the technical reality, as the defensive half of this industry isn't keeping pace with the offensive half, and the gap is only widening.
Offense in AI is cheap, quick to iterate, and endlessly inventive, and it also compounds on itself. Every model that ships just represents an additional surface to attack. The agent frameworks we're racing to adopt thread those models straight into systems that were never built to be argued with, and the reflex from product teams to bolt some vibecoded AI chatbot onto everything seems to be introducing new ground to attackers faster than defenders can survey what they already have. The capability is democratising at a speed we've simply not seen before.
I would also say the same models that make a defender marginally more productive now turn a mediocre script kiddie dangerous, and a capable hacker into something closer to a force multiplier (we might need a new AI acronym for APTs soon). We are, quite deliberately, arming both sides with better weapons, but only one of them is slowed down by the bureaucracy of change advisory boards, procurement cycles, and the desire not to break things in prod on the way through.
I see defense as something that moves the other way. It's still an expensive cost centre, deliberately slow, and inherits the weaknesses of everything it's attached to before adding a fresh, non-deterministic one of its own. We are threading LLMs through email, code, finance, infra, and increasingly into each another, faster than we can secure any single one of them, let alone the emergent behaviour of all of them working together.
From what I can see the industry's public posture on this seems to be leaning toward optimism. Its private posture, in my experience, is leaning closer to managed panic.
Where that leaves me
Overall my offsec research into AI this year has been enjoyable, but it concerns me. Yes, we've met new attack surfaces before (web, mobile, cloud etc) and eventually brought them under reasonable control, but the weaknesses behind them were deterministic, and deterministic problems are finite (so you can eventually design them out for good). The core problem with AI is neither, which is why the pocket of smug social media experts who've never hacked anything, yet dismiss those that have as alarmists, doesn't reassure me here.
If anything, it's pushed me the other way. I see a lot of concerns emerging lately from individuals with respectable offsec backgrounds, and I'm now siding with them. Even if that means having to defend my views online against random pockets of technically illiterate compliance clowns and AI slop bots.
I really think the people who are going to matter over the next few years are the ones who took the trouble to learn both halves of this and can sit in governance meetings and translate a framework into controls that can be validated, precisely because they've spent time on the other side watching those controls fail.
A lot of that comes down to language. The most valued people I've come across aren't purely technical or commercial - they're fluent in both, and can work a boardroom just as well as an architecture forum. Someone like Elon Musk is perhaps the obvious example, seemingly as comfortable steering a board as he is in the weeds with his engineers. That kind of bilingualism is rare, and I think it's what earns the respect of business people and engineers alike, because each side can tell when you actually understand theirs.
I've spent most of my career fluent in the language of engineers, and the last several deliberately working on my fluency in the language of business. I think the people who can speak both are the ones who can translate and interpret effectively between them.
There's also a more self-interested reason I've recently leaned this way. The AI boom is determining which jobs are safe, and the ones that look most exposed are the ones a lot of people can already do. Governance and risk management matter enormously, but my own sense is they're also crowded fields with a deep reserve of capable people who can fill those seats. Genuinely competent technical expertise in cybersecurity is scarcer. Done well, only a relatively small number of people can actually do it (even with AI enabled cognitive offloading), and rarer still is the person who can do that and also hold their own in front of an exec committee.
So the question I often encourage those concerned about immediate job security to ask themselves is a rather blunt one.
How many people can really do what you do?
For me, that answer has meant committing strongly to the overlap of both halves and recognising their equal importance. That intersection is where I've deliberately placed myself this year, and I'm going to work hard to entrench myself by continuing to upskill and further my AI research.
ABOUT THE AUTHOR
