The AI security question leaders should be asking instead

In this Help Net Security interview, Frederic Bull, Security Officer at Gremlin, talks about what AI means for security teams. The conversation covers why asking what data a model was trained on is only part of the picture, and why least privilege and access controls still matter for AI agents.

It also looks at how AI has narrowed the skill gap between attackers and defenders, how the team handled about nine times as many vulnerabilities with the same staff, and why hiring now favors people with the experience to catch confident but wrong AI output.

When you talk to other security leaders right now, what’s the one AI-related question you’re tired of answering, because everyone’s asking it and nobody’s asking the better question underneath it?

The question I hear more often than anything else is: “What data was this model trained on?” The underlying concern driving this is confidentiality. Companies want to ensure their intellectual property won’t become part of an AI’s general knowledge. That is a valid concern, and it would be folly to imply otherwise, but stopping there leaves gaps in a true layered security approach.

A far less frequently asked question is: “How do you maintain authority and integrity of the data in question.” And we have clear answers. For many years the principle of Least Privilege, achieved through Authentication & Access controls, has remained paramount to ensuring secure environments, and continues to be. Technologies such as session-based RBAC via OIDC/OBO and appropriate permission scoping for the context, ensure AI’s don’t exceed the capabilities expected of them. This has been the bedrock of enterprise security programs for decades, and it is just as relevant today.

By continuing to implement modern solutions like session-based RBAC and appropriately scoped permissions, we can guarantee that AI agents never exceed their technical capability. As agent-to-agent communications become more prevalent these strategies provide for more strict ‘human-in-the-loop’ architectures, ensuring operators are positioned to recognize and correct issues before damage occurs.

Walk me through what an attacker’s toolkit looks like differently now compared to three years ago. What’s new versus what’s just faster?

I don’t want to downplay the significance of increased speed. Ultimately that is how this risk presents itself in the real world, and speed is paramount. Even state actors are racing against clocks to find and exploit before others discover the flaws. We are seeing a sharp increase in the vulnerability reporting over the last year, NVD has had to reprioritize classifications, leaving a great number of reported vulnerabilities unclassified. They simply can’t handle the increased volume of reports.

However, the mechanism behind the elevated risk is not simply speed, it’s an erosion of the asymmetry of ability that the security industry has historically benefitted from. To find the conditions allowing for an exploit, design and test that exploit, and employ it, typically required a great deal of ability and/or expertise. AI has taken that asymmetry much closer to even.

Not only can those exploitable contexts be found more quickly, but the method to leverage that can be devised and implemented by those who otherwise lack the ability, and increasingly, faster than those with the ability. That is going to continue to be a more significant factor as the contextual data AIs receive increases and the models become better at recognizing these patterns. Discounting external factors, such as governmental controls, it’s quite possible that symmetry is achieved and the question becomes one of human nature. In essence a question of how large the red team is compared to the blue.

Security teams are famously understaffed. Has AI closed that gap for you, or has it just changed what the gap looks like?

Staffing is always a contentious subject, and that’s unlikely to change, but I do not find that headcount is the answer to the AI-driven risks we’re seeing. Here at Gremlin, we processed just over 9 times the number of vulnerabilities over the last year as the previous, while maintaining the same staffing levels over the period. That certainly left a gap in our process. Using AI tools, largely LLM’s and supporting harnesses, we were able to handle that increase without additional headcount while simultaneously reducing our TTR by just over 5%.

It was not trivial to achieve this throughput in the face of such a large increase in volume, and we had to react to it in a relatively short time. It took experienced human engineers to design the system to achieve that, leveraging what LLM’s provide to us, and to stay current as those systems evolve. But by embracing those same tools our adversaries were using, and investing in infrastructure around those tools, we have seen success thus far.

Is AI changing what you look for when you hire security talent? What skill matters more now than it did five years ago, and what matters less?

Yes, and I don’t think that will come as a surprise. Current AI systems are quite capable at producing on par with many entry level analysts and engineers, and more often than not only need an experienced operator to recognize confidently wrong outputs, and correct them. That takes a broad range of knowledge and experience with not just diverse technical systems, but the compliance and governance requirements driving many security programs. This raises the bar for all talent and creates a gap where it is increasingly difficult for new talent to gain the experience needed to operate these AI systems effectively.

There is also an overarching concern of generational processing, and the potential for model collapse when the majority, or all, inputs into AI systems are themselves generated by AI systems. Similar to the need for genetic diversity in a population, LLMs need input diversity that today requires human curation. Currently we think of this as separate problems, a labor problem and a data problem. However, if we look historically, labor was how fresh inputs were provided to the systems, and we appear to be minimizing them at a time when we may need them most.

What’s the most excited you’ve been about something in this space recently, something that isn’t just marketing language?

I think the impact on application level security is exciting. It’s allowed us to move security concerns much earlier in the SDLC process by having AI evaluate against a curated list of criteria and policies, surfacing concerns and giving guidance in real-time as opposed to at some bottlenecked review period. This allows developers to propose and implement solutions with review from specialists, often avoiding the need for additional architecture and design requirements filtering back down the pipeline to developers.

I no longer feel like I’m playing a catch-up game to stay current with changes. Many basic tasks that we previously depended on external development teams for are now internal side projects for my team because with some explicit review we know ‘the LLM can do this for us’, making a significant difference in our cadence.