Identity & Trust: The Foundation of Secure AI Systems
NeuralStack | MS — AI Security
Every security control eventually reduces to two questions: who is acting, and how much should they be believed. Access control answers the first question. Trust calibration answers the second. Conventional application security has spent three decades building mature answers to both – X.509 certificates, OAuth scopes, RBAC, and mutual TLS. AI systems break nearly all of it, not because the primitives are wrong, but because LLM-based systems introduce actors, channels, and delegation patterns that the underlying model of identity was never designed to represent.
This article argues that most of what we currently label "AI security" – prompt injection, excessive agency, RAG poisoning, insecure MCP tool use – is better understood as a single underlying failure: the collapse of identity and trust boundaries inside the LLM pipeline. Treating these as isolated vulnerability classes leads to isolated patches. Treating them as instances of one problem leads to an architecture.
Why Identity Breaks Down First
Traditional identity and access management assumes a small, enumerable set of principals – users and services – each with a durable credential and a fixed permission scope. An agentic AI system violates every one of those assumptions simultaneously:
Non-human identities multiply combinatorially. A single agent may hold delegated credentials for Gmail, a calendar, a vector store, and three MCP servers, each spun up per session, per task, or per sub-agent. There is no durable 1:1 mapping between "identity" and "credential" anymore.
The model itself is not a stable principal. The same weights, given the same system prompt, can be steered into acting on behalf of an attacker simply by what text arrives in its context window. Identity, in the IAM sense, is supposed to be something you are; for an LLM, in-context instructions can temporarily overwrite who it appears to be acting for.
Delegation is implicit and unaudited. When an agent calls a tool, it is asserting "the user wants this action performed." Nothing in the standard architecture cryptographically verifies that assertion – it is inferred from the model's own reasoning trace, which is precisely the artifact an attacker can manipulate.
The Instruction/Data Conflation Is a Trust Boundary Failure
Prompt injection is usually described as an input-validation problem. It is more accurately described as an identity failure: the model has no reliable mechanism to distinguish content that carries the authority of the principal (the user, the system operator) from content that is merely data about the world (a retrieved document, an email body, a tool response). OWASP formalizes this as LLM01:2025, and it retains the top spot in the 2025 revision for good reason — every other category on the list is, in some sense, downstream of this one.
Once you frame it as identity, the mitigation direction changes. Rather than trying to detect "malicious-looking" text, the goal becomes: can the system cryptographically or structurally distinguish who is speaking? Techniques worth taking seriously in this light include structured prompting with explicit provenance tags on every context segment, dual-model architectures where a privileged planner never sees raw untrusted content directly, and treating tool outputs as data-plane content that must be re-authorized before triggering further action – never as instructions.
Agentic Identity: Excessive Agency as an Authorization Failure
LLM06:2025, Excessive Agency, is where identity and trust collide most visibly in production systems. An agent with email-send permission and a prompt-injected instruction from an inbound message is not a prompt injection incident in isolation – it is an authorization failure: the system granted a standing permission scope broad enough that a single manipulated turn could exercise it.
The MCP ecosystem makes this concrete. Each MCP server the agent connects to represents a trust boundary the architecture must reason about explicitly:
Tool-level scoping, not agent-level scoping. An agent should hold the minimum viable permission for the current task, not a persistent superset "in case it's needed later."
Server provenance verification. An MCP server is code you did not write, running with the authority your agent grants it. Treat it the way you would treat a third-party dependency – because that is what it is.
Attestable tool calls. Every invocation of a write-capable tool (send, delete, transfer, execute) should be traceable to a specific upstream instruction with a verifiable chain back to the human principal, not just a plausible-looking reasoning trace.
RAG and the Provenance Problem
Retrieval-Augmented Generation reframes the same issue at the data layer. LLM04:2025 (Data and Model Poisoning) and LLM08:2025 (Vector and Embedding Weaknesses) both describe variants of a single question: when a document is retrieved and placed in context, what is the model actually trusting?
Most RAG pipelines answer implicitly: "if it's in the index, it's trustworthy enough to influence the answer." That is an identity claim with no verification behind it. A poisoned document injected into a shared vector store, or an embedding-space attack that causes semantically unrelated but adversarial content to surface for a benign query, both exploit the fact that retrieval conflates relevance with authority. A mature RAG security posture separates them: retrieved content is scored for relevance by the embedding model, but its influence on agent behavior (as opposed to informational output) should be gated by a separate, explicit trust label tied to source provenance – internal vs. external, signed vs. unsigned, and access-tier of the originating system.
Supply Chain: Trusting the Model Itself
LLM03:2025, Supply Chain, pushes the identity question one layer further back: before you ask whether an agent's outputs can be trusted, you have to ask whether the model and its dependencies can be. Fine-tuning datasets, third-party adapters, quantized community checkpoints, and even the training pipeline's own build dependencies are all potential injection points – a form of adversarial ML supply-chain compromise that maps directly onto MITRE ATLAS tactics around resource development and initial access via ML artifacts. An ML-BOM (bill of materials for model provenance – base weights, fine-tuning data lineage, adapter sources) is the identity credential for the model itself, and most organizations deploying LLMs today have no equivalent artifact at all.
Toward a Zero-Trust Architecture for AI Systems
Pulling these threads together, a coherent identity-and-trust architecture for AI systems rests on four practical commitments:
Provenance-tagged context. Every segment entering the context window – system prompt, user turn, retrieved document, tool output – carries an explicit, structurally enforced trust label. The model architecture, not the prompt wording, enforces the distinction.
Least-privilege, task-scoped agency. Permissions are granted per task, not per session or per agent identity, and expire with the task.
Verifiable delegation chains. High-impact actions (send, write, delete, execute, financial transfer) require a traceable authorization path back to a human principal – not just a coherent-sounding reasoning trace.
Continuous, not point-in-time, verification. Trust decisions made once at connection time (an MCP server was safe when reviewed) decay. Runtime monitoring – the deferred layer in a defense-in-depth scanner design – exists precisely because static trust assessments go stale as models, prompts, and upstream servers change.
Closing Thought
Every category in the current OWASP LLM Top 10 – prompt injection, excessive agency, supply chain, vector poisoning, system prompt leakage – reads differently once you stop treating it as a standalone vulnerability class and start treating it as a symptom of unresolved identity: the system does not have a reliable answer to "who is asking, and how much should that be believed" at every layer where a decision gets made. Building AI systems that are secure by architecture, rather than secure by patch, starts with making that question answerable – structurally, not heuristically – at the model, the agent, the tool, and the data layer alike.
NeuralStack | MS — AI Engineering & AI Security
