LLM observability and tracing

Why normal monitoring isn't enough

Normal application monitoring tells you a request took 4 seconds and returned 200. For an LLM app, that's not enough. A user says "the bot gave me the wrong price", and to find out why, you need to see whether retrieval found the wrong document, the prompt template dropped a field, the model ignored the context, or the pricing tool returned stale data.

Metrics, logs and traces

Signal	Answers	Examples
Metrics	How is the system doing overall?	Requests per minute, error rate, p95 latency, tokens, cost
Logs	What events happened?	Errors, guardrail triggers, fallbacks
Traces	What happened in this request, step by step?	Spans for each model call, retrieval and tool call

All three matter, and for LLM apps, traces are where most debugging happens.

What to capture

The full list is in what a production trace records. The minimum per request:

• Trace ID, session ID, and user or tenant ID (hashed if needed).
• Each span: type, start, end, status, error.
• Model spans: model and version, the rendered prompt, the output, tokens, finish reason, cost.
• Retrieval spans: query, results with scores, chunks used.
• Tool spans: name, arguments, result, error.
• The prompt version and app version.
• Feedback events, linked to the trace.

Mask personal data before it reaches your tracing backend, and set a retention period that matches your policies.

Tools

• OpenTelemetry: the open standard for traces, metrics and logs, with GenAI semantic conventions for LLM spans (still in development).
• Langfuse: open source, self-hostable, with tracing, prompt management, evals and cost tracking.
• Arize Phoenix: open source, with tracing and evaluation, built on OpenTelemetry.
• LangSmith: managed platform from the LangChain team, with tracing and evaluation.
• OpenLLMetry: OpenTelemetry instrumentation for LLM libraries.
• Prometheus and Grafana: metrics and dashboards for the service around the model.

Choose one tracing backend, and instrument through OpenTelemetry where you can, so you can switch later.

A minimal setup

1. Instrument the request entry point, so every request gets a trace ID.
2. Wrap each model call, retrieval call and tool call in a span with the fields above.
3. Send traces to your backend, and metrics to your dashboards.
4. Add the trace ID to error messages and support tickets, so a complaint leads straight to the trace.
5. Open 10 traces. If you can't tell what happened in each one, add the missing fields.

import json
import time
import uuid
from contextlib import contextmanager

spans: list[dict] = []


@contextmanager
def span(trace_id: str, name: str, **attrs):
    record = {"trace_id": trace_id, "span": name, "start": time.time(), **attrs}
    try:
        yield record
        record["status"] = "ok"
    except Exception as exc:
        record["status"], record["error"] = "error", str(exc)
        raise
    finally:
        record["duration_ms"] = round((time.time() - record["start"]) * 1000, 1)
        spans.append(record)


trace_id = str(uuid.uuid4())
with span(trace_id, "retrieval", query="refund policy electronics") as s:
    s["chunk_ids"] = ["policy-2025-eu#4", "policy-2025-eu#5"]
with span(trace_id, "chat", model="small-model-2026-08", prompt_version="refund-v7") as s:
    s["input_tokens"], s["output_tokens"], s["finish_reason"] = 1820, 96, "stop"

for record in spans:
    print(json.dumps({k: v for k, v in record.items() if k != "start"}))

In practice, a tracing SDK does this for you, and the shape of the data is the same.

What to watch

• Share of requests with complete traces.
• Error and fallback rates.
• p95 latency per stage.
• Tokens and cost per request and per route.
• Guardrail trigger rates.
• Feedback rates (thumbs down, retries, handoffs).
• Median time from a user complaint to finding the cause.

From the floor

In DevFix, we added observability so that every node of the LangGraph agent, every LLM call and every Docker execution was traced with its latency and cost, first with LangSmith and later with Langfuse, which is open source and self-hostable. When a builder asked how LangChain, LangGraph and LangSmith relate, my answer was: LangChain for working with LLMs, LangGraph for building agents, and LangSmith or Langfuse for observing every step of the execution.

In the production pipeline I teach, the last zone ends with tracing and continuous evaluation before the answer returns to the user, with LangSmith as the example there, because that dashboard isn't for the user, it's for us, the people building the system.

Related

• What a trace records
• Cost monitoring
• Design LLM observability
• Audit logs

Sources

• OpenTelemetry semantic conventions for generative AI
• Langfuse documentation
• Arize Phoenix documentation
• LangSmith documentation