Collecting user feedback in AI products
Collect explicit feedback (thumbs, ratings, comments, choosing between two answers) and implicit feedback (edits, copies, retries, abandonment, escalation to a person, task outcomes), always linked to the trace ID. Implicit signals are far more plentiful, and explicit ones are clearer, so you need both. What users do after an answer says more than what they rate.
Why thumbs aren't enough
Most users never click thumbs up or down. The ones who do are a small, biased slice: the very happy and the very annoyed. If thumbs are your only signal, you see a sliver of what's happening.
Signals and what they mean
| Signal | Type | What it usually means |
|---|---|---|
| Thumbs up or down | Explicit | A direct opinion, but rare |
| Comment | Explicit | Often the exact problem, but rarer |
| Choosing A over B | Explicit | Preference data |
| Edit before sending | Implicit | The draft was close, and the edit shows what was wrong |
| Copy | Implicit | The answer was useful |
| Retry or rephrase | Implicit | The first answer missed |
| Abandonment | Implicit | Gave up, or got the answer and left, so check the trace |
| Escalation to a person | Implicit | The assistant failed, or the user lost trust |
| Task completed | Outcome | The result you care about |
Link everything to traces
A thumbs down without a trace ID tells you something went wrong somewhere. With a trace ID, you can open the exact request, see what the model saw and what it said, and turn it into an eval case.
import json
import time
import uuid
def feedback_event(trace_id: str, kind: str, value=None, user_id: str = "u_381") -> dict:
return {
"event_id": str(uuid.uuid4()),
"trace_id": trace_id,
"kind": kind,
"value": value,
"user_id": user_id,
"ts": int(time.time()),
}
trace = "tr_8f3a91"
events = [
feedback_event(trace, "thumbs", "down"),
feedback_event(trace, "edit", {"before": "within 30 days", "after": "within 14 days"}),
feedback_event(trace, "escalated", True),
]
for e in events:
print(json.dumps({k: e[k] for k in ("trace_id", "kind", "value")}))
Design the product to collect it
- Put feedback controls where they're easy and don't interrupt the task.
- Let users edit drafts instead of only accepting or rejecting them, and store both versions.
- Ask for a short reason only when it's a thumbs down, and make it optional.
- Track outcomes from the business system (ticket resolved, invoice approved), not only in the chat.
Using it
- Weekly: review the 20 worst-rated traces, and the traces with edits or escalations.
- Turn failures into eval cases.
- Use edits and A-or-B choices as preference data for DPO, once you have enough.
- Watch feedback rates per release as an early warning.
Privacy
Feedback is user data. Tell users how it's used, mask personal data before it goes into eval or training sets, and respect contracts that limit how customer data can be used.
Measuring it
- Share of traces with any feedback signal.
- Eval cases created from feedback this month.
From the floor
In the customer support agent, the handoff to a person was triggered when the router's confidence was low, and the escalated query was posted to a Discord channel through a webhook, which is also a feedback signal: every escalation is a trace where the assistant couldn't handle the request, and those are the first traces to read.