Documentation, 231 pages
Build production AI systems, one problem at a time
Everything you need from your first LLM call to the architecture of a full inference pipeline. You don't read these in order. Pick a problem, open the pages it needs, build something, and come back.
Pick a track
01
Engineer
Moving into AI engineering. LLM fundamentals, then retrieval, agents, evals and deployment.
02System design
Senior engineers and architects. Start with the production inference pipeline, then patterns and reference architectures.
03Leaders
Heads of AI, CTOs and VPs. Start with why AI projects stall between pilot and production.
Contents
0. Start here
Free docs on AI engineering. Start with a problem, open the pages your build needs, build it, and come back when the next part asks for more.
1. Builder foundations
The software skills every AI build depends on: Python, APIs, async, data formats, Git, the terminal, Docker, cloud deploys, secrets and validation.
- 1.1Python for AI engineers: what you actually use
- 1.2Working with LLM APIs: retries, rate limits, timeouts and streaming
- 1.3Async Python for LLM apps
- 1.4Data formats for AI work: JSON, JSONL, Parquet and embeddings on disk
- 1.5Git and GitHub for AI projects
- 1.6Linux and the terminal for AI work
- 1.7Docker for AI apps
- 1.8Cloud basics for AI engineers: your first deploy
- 1.9Environment variables and secrets management for AI apps
2. LLM fundamentals
How LLMs work for people who build with them: tokens, context windows, sampling, embeddings, model choice, cost, reasoning, tools and hallucination.
- 2.1How an LLM works, explained without math
- 2.2Tokens and tokenization: why LLMs can't count the r's in strawberry
- 2.3Context windows: what fits, and why more isn't always better
- 2.4Sampling: temperature, top-p and top-k explained
- 2.5Why LLMs give different answers at temperature 0
- 2.6Embeddings: what they are and what they are not
- 2.7LLM model families: closed APIs, open-weight, small and reasoning models
- 2.8How to choose an LLM for your product
- 2.9LLM pricing math: cost per request, per user and per month
- 2.10Reasoning models: when the extra latency and cost pay off
- 2.11Multimodal models: vision, audio and documents
- 2.12Structured outputs and function calling explained
- 2.13LLM hallucination: causes and the controls that work
3. Prompt and context engineering
How to write prompts that hold up in production and decide what the model sees on every call: system prompts, examples, context, caching and testing.
- 3.1Prompt engineering that survives production
- 3.2Context engineering: deciding what the model sees on every call
- 3.3System prompts and instruction hierarchy
- 3.4Few-shot examples: how to select and order them
- 3.5Long context vs retrieval (RAG): when to use which
- 3.6Prompt templating and versioning
- 3.7Prompt caching: cut cost and latency on repeated context
- 3.8Context compression and summarisation
- 3.9Prompt testing before release
4. Retrieval and RAG
How a production RAG system works: ingestion, parsing, chunking, embedding, indexing, retrieval, reranking, generation with citations, and evaluation.
- 4.1Document parsing for RAG: PDFs, tables and scanned pages
- 4.2Chunking strategies for RAG and how to choose chunk size
- 4.3How to choose an embedding model for RAG
- 4.4Vector databases for RAG: what they do and how to pick one
- 4.5Vector index internals: build a vector store in pure Python, then HNSW, IVF and PQ
- 4.6Hybrid search for RAG: combining BM25 and vector search
- 4.7Reranking in RAG: why and how to add a cross-encoder
- 4.8Query rewriting and decomposition for RAG
- 4.9Metadata filtering and access control in RAG
- 4.10GraphRAG and knowledge graphs: when graphs beat plain vector search
- 4.11Agentic RAG: when retrieval needs to plan
- 4.12How to evaluate a RAG system: retrieval metrics vs answer metrics
- 4.13RAG failure modes: a catalogue with fixes
- 4.14RAG for regulated data: healthcare, pharma, finance and defence
5. Agents and tool use
An AI agent uses an LLM to choose actions, calls tools, reads results and repeats until done. How agents work, and when a script or workflow is better.
- 5.1Workflows vs agents: how to decide
- 5.2Tool design for LLM agents
- 5.3Model Context Protocol (MCP) explained: servers, clients and when to use it
- 5.4Agent planning patterns: ReAct, plan-and-execute and reflection
- 5.5Agent memory: short-term, long-term and episodic
- 5.6Multi-agent systems: when they help and when they hurt
- 5.7Agent frameworks: LangGraph, CrewAI, Microsoft Agent Framework and no-code tools
- 5.8Human in the loop for AI agents: where to put approvals
- 5.9Agent state, checkpoints and recovery
- 5.10Computer-use and browser agents
- 5.11How coding agents are built, and how to work with them
- 5.12Voice agents: architecture, latency and turn-taking
- 5.13Agent cost control and loop limits
6. Model adaptation
The ways to adapt an LLM to your task, from cheapest to most involved: prompting, retrieval, fine-tuning, distillation and quantization, and how to choose.
- 6.1When to fine-tune an LLM, and when not to
- 6.2Fine-tuning methods: full fine-tuning, LoRA and QLoRA
- 6.3Preference tuning: DPO and related methods
- 6.4Distillation: teaching a small model to do a big model's job
- 6.5Quantization explained: fp32, fp16, int8 and 4-bit models
- 6.6How to build a fine-tuning dataset
- 6.7How to evaluate a fine-tuned model
- 6.8Serving fine-tuned models and LoRA adapters
7. Evals and reliability
A practical evals playbook for LLM apps, RAG and agents: find failures in traces, measure them with validated checks, and run evals in CI and production.
- 7.1Why MMLU and GPQA scores don't tell you if your app works
- 7.2What a production trace records, span by span
- 7.3How to find your LLM app's failure modes in 30 to 100 traces
- 7.4Why one domain expert beats five annotators for LLM evals
- 7.5How to pick which LLM traces to read
- 7.6Why pass or fail beats a 1 to 5 score for LLM evals
- 7.7How to check if your LLM judge is right
- 7.8How to stop an LLM judge prompt from overfitting
- 7.9Why BERTScore, ROUGE and generic "helpfulness" scores mislead you
- 7.10How to test if an LLM knows when to say "I don't know"
- 7.11How to evaluate RAG in two separate halves
- 7.12How to choose chunk size with evals
- 7.13How to evaluate a coding agent
- 7.14How to evaluate an AI agent end to end and step by step
- 7.15How a transition failure matrix finds the broken step in an agent
- 7.16How to evaluate multi-turn conversations
- 7.17Guardrails vs evaluators: what runs inline and what runs later
- 7.18Why you should build your own annotation tool for LLM evals
- 7.19How CI gates and production monitoring work together for LLM apps
- 7.20When to retire an LLM eval
- 7.21Eval-driven development for LLM applications
- 7.22How to build an eval set from production traffic
- 7.23Online evaluation and A/B tests for LLM features
- 7.24Reliability patterns for LLM apps: retries, fallbacks, timeouts and circuit breakers
8. LLMOps and production
Ship and run LLM systems: the production checklist, inference serving and optimisation, caching, routing, tracing, cost, CI/CD, rollouts and incidents.
- 8.1From notebook to production: the LLM app checklist
- 8.2LLM inference serving: hosted APIs vs self-hosted models
- 8.3LLM inference optimization: prefill, decode, KV cache, batching and speculative decoding
- 8.4Latency budgets and streaming UX for LLM apps
- 8.5Semantic caching for LLM apps
- 8.6Model routing and LLM gateways
- 8.7LLM observability and tracing
- 8.8Cost monitoring and FinOps for AI applications
- 8.9CI/CD for AI applications
- 8.10Rollouts, canaries and model upgrades
- 8.11Incident response for AI systems
- 8.12Data flywheels: how AI products learn from production
- 8.13How to run LLMs locally and on edge devices
9. AI system design
A seven-step framework for designing AI systems, from business objective to monitoring, with the components, patterns and reference architectures to use.
- 9.1The LLM inference pipeline: a production architecture in six zones
- 9.2LLM systems architecture: chapter index
- 9.3AI system design patterns catalogue
- 9.4Guardrail sandwich pattern: checks before and after the model
- 9.5Hybrid fleet pattern: APIs, open models and fine-tuned models behind one router
- 9.6Tool registry and sandbox pattern for AI agents
- 9.7Exact values pattern: stop the LLM from rewriting your numbers
- 9.8Fallback chain pattern for LLM systems
- 9.9Async job queue pattern for long AI tasks
- 9.10Reference architecture: AI customer support assistant
- 9.11Reference architecture: in-product AI copilot
- 9.12Reference architecture: enterprise knowledge engine (on-prem)
- 9.13Reference architecture: AI document processing pipeline
- 9.14Reference architecture: multi-agent platform
- 9.15Reference architecture: voice assistant
- 9.16Non-functional requirements for AI systems
- 9.17Capacity planning and cost modelling for LLM systems
- 9.18Multi-tenant AI platforms: isolation, limits and per-tenant customisation
- 9.19Data architecture for AI systems
10. Security, safety and governance
Threat model, prompt injection, data leakage, access control, agent permissions, red teaming, compliance, audit logs and a responsible AI checklist.
- 10.1Threat model for LLM applications (OWASP Top 10 for LLMs 2025)
- 10.2Prompt injection: direct and indirect attacks, and what helps
- 10.3Data leakage and PII handling in LLM applications
- 10.4Access control for RAG and AI agents
- 10.5Agent permissions and sandboxing
- 10.6How to red team an AI application
- 10.7AI compliance basics: privacy law, sector rules and the EU AI Act
- 10.8Audit logs and traceability for AI systems
- 10.9Responsible AI checklist for shipping teams
11. Data for AI
The data work behind AI systems: ingestion pipelines, data quality for retrieval, labelling, synthetic data, feedback collection and dataset versioning.
12. AI data structures and algorithms
DSA for AI Engineers: a 12-module roadmap from classical DSA to tensors, embeddings, similarity search, KV cache, decoding, agent graphs and serving.
- 12.1From discrete to continuous values: why AI needs vectors, not hash keys
- 12.2Tensors and shapes for AI engineers
- 12.3Floating point precision and memory in AI models
- 12.4Logits and softmax: how a model turns scores into the next token
- 12.5Tokenizer data structures: byte pair encoding from scratch
- 12.6Embedding tables: from token IDs to vectors
- 12.7Nearest neighbour search: brute force, heaps and clustering
- 12.8KV cache: why LLM memory grows with context length
- 12.9Compute graphs: how deep learning frameworks track operations
- 12.10Hashing and caching for LLM apps: exact-match and LRU caches
- 12.11Queues and batching for AI workloads
- 12.12Graphs for agents and knowledge: state machines, DAGs and knowledge graphs
13. Career and interviews
Portfolio, resume, the AI system design interview with 25 worked questions, switching from backend or data roles, freelancing and alumni stories.
- 13.1AI engineer portfolio: what hiring managers actually look at
- 13.2AI system design interview: how to prepare, with 25 worked questions
- 13.3AI system design interview: design a customer support agent
- 13.4AI system design interview: design RAG over internal company documents
- 13.5AI system design interview: design a semantic cache
- 13.6AI system design interview: design a model router
- 13.7AI system design interview: cut your LLM bill in half
- 13.8AI system design interview: reduce latency for a chat assistant
- 13.9AI system design interview: design a document extraction pipeline
- 13.10AI system design interview: design a code review agent
- 13.11AI system design interview: design text-to-SQL for business users
- 13.12AI system design interview: handle PII in an LLM application
- 13.13AI system design interview: prevent prompt injection in an email agent
- 13.14AI system design interview: evaluate a RAG system before launch
- 13.15AI system design interview: design multi-tenant RAG for a SaaS product
- 13.16AI system design interview: deploy an LLM on-prem with no internet
- 13.17AI system design interview: Q&A over annual financial reports
- 13.18AI system design interview: design a legal citation verifier
- 13.19AI system design interview: design a multilingual assistant
- 13.20AI system design interview: design long-term memory for an assistant
- 13.21AI system design interview: keep session memory for 10,000 returning users
- 13.22AI system design interview: detect and reduce hallucination in production
- 13.23AI system design interview: design a voice agent for appointment booking
- 13.24AI system design interview: design AI search for an e-commerce catalogue
- 13.25AI system design interview: design a meeting summariser with action items
- 13.26AI system design interview: design LLM observability for a team
- 13.27AI system design interview: fine-tuning or RAG for a domain assistant?
- 13.28AI engineer resume: what to put on it
- 13.29Moving from backend or data engineering into AI engineering
- 13.30Freelancing and consulting as an AI engineer
- 13.31AI Engineer HQ alumni stories
14. AI leadership handbook
For leaders who own AI budgets and teams: strategy, use case selection, build vs buy, cost and ROI, pilots to production, teams, governance and reporting.
- 14.1Why companies aren't shipping the AI they already bought
- 14.2AI readiness and maturity assessment for your organisation
- 14.3AI strategy on one page
- 14.4How to pick your first three AI use cases
- 14.5Cloud APIs vs open-source LLMs: the leader's decision
- 14.6Build vs buy vs partner for AI capabilities
- 14.7Estimating AI ROI and cost before you commit
- 14.8From POC to production: why AI pilots die and how to avoid it
- 14.9Structuring an AI team: roles and ratios
- 14.10Hiring AI engineers: what to test for
- 14.11Upskilling your existing engineers for AI
- 14.12AI governance without killing velocity
- 14.13AI vendor evaluation scorecard
- 14.14Measuring AI adoption inside the company
- 14.15Board-level AI reporting
- 14.16Head of AI's first 90 days
15. Glossary
Plain definitions of AI engineering terms, from agents and attention to vector databases and zero-shot prompting, with links to the full pages.
16. Comparisons
Side-by-side comparisons of agent frameworks, vector databases, LLM API providers, inference servers, RAG frameworks and key design choices.
- 16.1LangGraph vs CrewAI vs Microsoft Agent Framework (2026)
- 16.2pgvector vs Qdrant vs Pinecone for RAG (2026)
- 16.3LLM API providers compared: OpenAI vs Anthropic vs Google (2026)
- 16.4vLLM vs SGLang vs Ollama: which LLM inference engine (2026)
- 16.5LlamaIndex vs LangChain (2026)
- 16.6RAG vs fine-tuning: which one does your problem need?
- 16.7MCP vs function calling: what's the difference?
- 16.8Hosted LLM APIs vs self-hosted models: engineering comparison
17. Teardowns
Architecture teardowns of familiar AI products, reasoned from public behaviour and documentation. Clearly labelled as inference, not insider knowledge.
18. FAQ
Questions builders and leaders asked in AI Engineer HQ, Build30 and workshop sessions, with Himanshu's answers and links to the full docs pages.
Frequently asked questions
Questions builders and leaders asked in sessions, grouped by topic. Each one jumps to its answer on the FAQ page.
Career and learning
- Q: Are an ML engineer and an AI engineer the same role?
- Q: Should I start with a harder project, or go back and finish the simpler ones first?
- Q: Do I need Claude Code, Copilot and Codex, or is one coding assistant enough?
- Q: Do AI engineering interviews focus on algorithms?
- Q: Do certifications help me get an AI engineering job?
- Q: Should I go for a job or freelancing first?
- Q: How do I find freelance clients?
- Q: Do I need many followers before LinkedIn shows my posts to more people?
LLM fundamentals
- Q: What exactly is a large language model?
- Q: Does ChatGPT remember everything I've told it?
- Q: Why does an LLM give a different answer when I ask the same question again?
- Q: Is a different answer to the same question a hallucination?
- Q: Can we tell the GPU to process things in a fixed order so the output is always the same?
- Q: Why can't the LLM calculate this? Isn't it simple math?
- Q: Do I need to fine-tune a model to do grammar correction or summarisation?
- Q: How many transformer blocks are there in a model?
- Q: What's the difference between an encoder and a decoder?
- Q: At the first embedding step, does a token know anything about the sentence around it?
RAG and embeddings
- Q: How is RAG different from an LLM?
- Q: Is a query vector the same as an embedding vector?
- Q: Does every file, Excel, CSV, PDF or Word, have to become vectors?
- Q: What does a negative number in an embedding mean?
- Q: What is cosine similarity?
- Q: Is ChromaDB a vector database?
- Q: What chunking strategy should I use for a small dataset?
- Q: My support content is HTML pages, not PDFs. Can I still use it?
- Q: Can I deploy the customer support agent on my company's own data?
Agents
- Q: If the LLM can calculate, where does the agent come in?
- Q: Is an agent just a workflow?
- Q: What do "tools" and "thinking" mean for an agent?
- Q: Is LangChain only for LLM apps, so I need LangGraph for agents?
- Q: Why did we install Postgres for the agent?
- Q: Does a skill file I write for Claude Code also work with Copilot or other assistants?
- Q: Is n8n free to host?
APIs, tools and setup
- Q: What is the API key for?
- Q: Does an OpenAI API key work with other providers' models?
- Q: Can my Python agent call an existing API written in .NET or Java?
- Q: If I use LangChain, do I still need TensorFlow?
- Q: Do I still need a cloud provider if I build my agent with LangChain or Claude Code?
- Q: Where can I get a free API key to practise with?
- Q: Do I have to buy Azure Machine Learning to use it?
- Q: What is a pull request for?
- Q: I have Python 3.10, 3.11 and 3.13 installed. Which should I keep?
- Q: Should I add comments to my code?
- Q: Should I clear or compact the context in Claude Code between sessions?
Fine-tuning, quantization and local models
- Q: Why fine-tune if we can already use RAG?
- Q: If I download and run a model locally, does my data still go to a public model?
- Q: How much accuracy do I lose when I quantize a model?
- Q: How do I check the quality of a quantized model's output?
- Q: Does all quantization happen through llama.cpp?
- Q: What is Unsloth?
- Q: Did Bloomberg use Ollama for BloombergGPT?
- Q: Do I need a GPU, or can I use Google Colab?
Production and system design
- Q: Why put an API gateway in front of the LLM?
- Q: Does a request always go to the cache first, then the vector database?
- Q: Is the cache prompt caching or semantic caching?
- Q: How does the router decide whether a query is simple or complex?
- Q: Doesn't adding an intent router increase latency?
- Q: How do you manage session memory for 10,000 users who come back a week later?
- Q: Should we go cloud, on-prem, or both?