Welcome back. This is Part 2 of our series on how to actually architect security for LLMs and agentic artificial intelligence systems in the real world. If you missed Part 1, here’s the short take: LLM security is not just application security. It can’t be trusted to tell an instruction from data. Any document it reads is an attack vector. Or throw a filter on top? That will never give a regulated business the certainty it needs. So if the model cannot be your security boundary, then what can? Most enterprises are discovering the AI gateway. It’s a deterministic, policy-enforcing layer; a real piece of infrastructure, rather than a clever prompt or a “safer” model. That layer sits between your LLM and the rest of your world, and critically, it doesn’t trust the model’s judgment. Because it shouldn’t be.
Why Just “Add a Gateway” Won’t Do It
If you’ve built API gateways, “AI gateway” probably sounds familiar—rate limits, auth, routing, done this before, right? Yes and no.
Regular API gateways care about the shape of a request, is this caller authenticated, are they within their rate limit, does the payload fit the schema. They do not really care or even know what the request means.
But an AI portal? It has to do all that and something most gateways have never even considered about, make policy decisions about intent. What does this model want to do? Who’s data? Who for, and on what terms? Is this current action permissible for this specific call? That’s a different job entirely. It’s not enough to just point your API gateway config at an LLM endpoint and be done with it.
This is what the big picture looks like:
Please note that the model is behind the security boundary — not at the boundary. It’s just a reasoning engine. The gateway is the one deciding what’s actually allowed to happen.
The Policy Firewall: Intent Meets Authorization
The policy firewall: the heart of the new approach. This is between “the model thought of something” and “the thing is happening.”
This is critical when your LLM starts taking real actions – calling tools, hitting APIs, writing to databases. In Part 1, I highlighted the distinction between a chatbot that responds to questions, and an agent that can write to your systems. It is the policy firewall that makes that distinction turn into action.
Let’s dissect a single tool call through the firewall:
The point is, none of these checks care if the model was “well-behaved”, or if the prompt was maliciously crafted three steps back. The firewall is not judging the logic of the model, it is judging the action through a security lens, every single time.
That’s it. You’re not trying to detect all the attacks, you’re trying to make the attacks irrelevant. Even if it goes rogue, the model can’t do anything outside of explicitly authorized policy.
Identity must be passed down to each tool call
Today, most enterprise AI setups authenticate once at the beginning of a session and then let the agent roam with a broad credential. (Yes, it’s easier, but it’s also dangerous — see Part 1 for the horror stories.)
The answer? Push identity enforcement down into each tool call:
This is the difference between an agent with broad access and an agent that can only invoke access, one call at a time, under a traceable identity. This gives you an audit trail when something goes wrong (and it will). The old way? Just a shrug and a mystery.
Input & Output Inspection: Load-Bearing, Not Optional
But as mentioned in Part 1: filters are not a security boundary, on their own. But inside a gateway, input and output inspection is an important control – not your only defense, but a necessary part of the puzzle.
Pre-model inspection: catches what it can before the prompt hits the model – known attack patterns, sensitive data, untrusted content.
Pre-downstream inspection: You catch risky output before it goes downstream, not all model output is safe just because it is from the model.
Neither filter has to be perfect. Why? Because they aren’t your only line of defense anymore. We’re not “hoping” the filter holds, we’re designing so that
if it does fail, the rest of the architecture picks up the slack.
What It Costs (And Why It Is Worth It)
So this is not free. A policy firewall that does per-tool-call identity checks is expensive in latency, infrastructure, and real engineering time. “You can’t get this by ‘just calling the model API'”
But what else can you do? An AI agent that has permanent access to your most sensitive systems, shielded only by a system prompt that says “please don’t do anything harmful.” Gateway architecture is not overhead. It’s the difference between a contained and logged event and a full-blown breach.
Coming Up in Part 3: Data Boundaries and the RAG Attack Surface
Part 1 outlined the problem. This post described the enforcement layer surrounding the model. But the next big question is: So, when RAG (Retrieval-Augmented Generation) enters the chat, how do you actually control what the AI is allowed to retrieve, where from, and with whose permissions?
Next up in Part 3, document-level access control, provenance tracking, and why “the AI has read access to the shared drive” should worry every security architect.
This is part 2 of a 4-part series on LLM and agentic AI security architecture. Follow @VamsiTalksTech for the next update.
Featured image designed by Freepik