Security, safety and governance for LLM applications
LLM applications add new attack surfaces to ordinary software: text that acts like code, models that can repeat what they've read, and agents that can take actions. Secure them the way you secure any system (least privilege, access control, logging, review) plus the controls specific to LLMs: treating every piece of external text as data, masking personal data before it reaches a model, limiting what agents can do, and testing with attacks before release. Most of the work is design, not filters: decide what the AI can read, what it can do and who it acts for, and enforce that in code.
What's different about LLM security
In normal software, code and data are separate. A database query is code, and the customer's name inside it is data, and if you escape the data properly, it can't become code.
In an LLM application, instructions and data arrive as the same thing: text. The system prompt, the user's question, a retrieved document and a tool result all go into one context, and the model decides what to follow. That is why prompt injection exists, and why no filter fully solves it.
Three properties create most of the risk:
- The model reads untrusted text (user messages, web pages, emails, documents).
- The model can see sensitive data (retrieved documents, customer records, the system prompt).
- The model can act (tools that send, write, buy or delete).
When all three are present in one system, an attacker who controls some text can try to make the model leak data or take actions. Remove any one of them for a given flow, and the risk drops sharply.
Safety and governance
Security is about attackers. Safety is about harm even without an attacker: wrong medical advice, a biased decision, a confident made-up citation. Governance is how an organisation decides what AI it builds, who approves it, how it's monitored and who is accountable. All three show up in the same places in an architecture: inputs, outputs, permissions and logs.
Where the controls sit
| Layer | Controls | Page |
|---|---|---|
| Input | PII masking, injection detection, scope checks | Prompt injection, Data leakage and PII |
| Retrieval | Permission filters, tenant isolation | Access control |
| Tools | Least privilege, sandboxing, approvals | Agent permissions |
| Output | Validation, leakage and safety filters | Guardrail sandwich |
| Process | Threat model, red teaming, compliance, audit | Threat model, Red teaming, Compliance, Audit logs |
What this section covers
| Page | The question it answers |
|---|---|
| Threat model | What can go wrong in my LLM app? |
| Prompt injection | How do I limit the damage from instructions hidden in text? |
| Data leakage and PII | How do I keep personal data out of prompts and logs? |
| Access control for RAG and agents | How do I make sure the AI sees only what the user can? |
| Agent permissions and sandboxing | How much should an agent be allowed to do? |
| Red teaming | How do I attack my own system before others do? |
| Compliance basics | Which laws and rules apply? |
| Audit logs | Can I explain any answer months later? |
| Responsible AI checklist | Is this feature ready to launch? |
From the floor
When I explain why some organisations won't use public AI APIs, the examples come from the sectors we've worked with, hospitals, law firms and defence: a hospital can't send a patient's medical reports to ChatGPT, because that would violate the rules it operates under, so for these industries privacy and security come first and everything else is secondary.
In Neural Vault, the AI Engineer HQ artifact for private knowledge, the target users were what I called zero-trust professionals, people who won't trust any external AI provider with their data, so the design started from those constraints: fully offline, air-gapped, with PII masking and audit logs.