Observability for AI agents in a retail workflow
Trace a retail AI agent’s question, source records and tool outcomes so you can diagnose a wrong answer or failed action without exposing customer data.
- Published
- Last updated
Observability for AI agents is the ability to follow a task through the records the agent reads, the tools it calls and the outcome the customer receives. A useful trace lets you locate where an answer or action went wrong. It also separates an attempted action from a change that actually saved.
For a retailer, the starting question is practical: why did the agent say a return was created when the order has no return? Follow that one task before changing the agent’s instructions or retrying the action.
Trace one retail task
A task trace joins the customer’s request to the evidence and actions used to answer it. Each operation has its own record, with a shared task identifier connecting the sequence. A trace shows recorded behaviour. It cannot reveal every hidden internal model thought.
Consider this fictional return request. A customer wants to return an unopened kettle from order ORD-204. The store permits this return, but the agent tells the customer it has created one before the returns service finishes processing the request.
The following trace uses invented identifiers and elapsed times. It contains only the information needed to diagnose that claim.
| Elapsed time | Recorded operation | Evidence or result |
|---|---|---|
| 0 ms | Task T-81 begins | Request to return kettle line L-2 from ORD-204 |
| 120 ms | Order lookup completes | L-2 is product K-40, quantity one, delivered and not already returned |
| 210 ms | Policy lookup completes | Returns policy version 7 permits this unopened item |
| 350 ms | Return tool receives request A-19 | Order ORD-204, line L-2, quantity one and reason unopened |
| 510 ms | Return tool responds | Request accepted for processing, job J-9 pending |
| 620 ms | Customer answer appears | “Your return is created.” |
| 1,400 ms | Job J-9 completes | Failed execution, no return record created |
The order lookup is source evidence. Request A-19 records what the agent asked the tool to do. The pending response and later failure record what the service returned. The customer’s answer is a separate event that must agree with those results.
Keep source identifiers and versions alongside the values used. A catalogue entry can change after a conversation. For a product-choice task, retain the selected product and variant identifiers, the relevant stock snapshot and the catalogue version. Record when each operation starts and ends so you can identify a delayed result.
Separate the failure modes
Diagnose the first point where the evidence stops supporting the next step. Similar customer complaints can require different corrections, so compare the source and action records before changing the final answer.
| Failure mode | Evidence to inspect | Focused correction |
|---|---|---|
| Wrong source data | The retrieved product or order identifier differs from the requested item | Correct record selection or the source mapping |
| Stale policy | The retrieved policy version differs from the applicable approved version | Refresh the policy source and its retrieval path |
| Unsupported tool arguments | The request contains an unknown line identifier or disallowed quantity | Validate arguments against the tool contract before sending |
| Permission refusal | The service denies the requested operation for this customer or agent | Correct the authorised access path or hand the case to staff |
| Failed execution | The request passes validation but the operation ends in an error | Resolve the service failure and establish saved state before retrying |
| Overstated answer | The customer hears “created” while the result remains pending or failed | Tie the completion message to confirmed completion |
In task T-81, policy retrieval supports eligibility and the tool accepts the request. Neither fact supports “Your return is created”. The service’s final result shows a failed job with no saved return. The answer overstates the outcome, and execution also needs investigation.
A successful request can mean only that a service received or queued work. For a completed return, require a return identifier and the expected saved status from the system that owns returns. If the write response doesn’t establish that state, perform an authorised lookup using the operation identifier.
When the result is uncertain, tell the customer that the request is still being checked. A missing response is insufficient evidence for either success or failure. Look up the first request before issuing a second one, especially for payments or order changes.
Keep the trace usable and controlled
Choose trace fields from the diagnostic question you need to answer. For task T-81, you need to connect the return claim to the tool result. You don’t need the customer’s entire shopping history or unrelated messages.
Use a small event contract that your team can interpret consistently.
| Field group | What to retain | What it helps establish |
|---|---|---|
| Correlation | Task identifier, event identifier and parent operation identifier | Which events belong together, including asynchronous work |
| Execution context | Workflow version, agent configuration version and model identifier | Which configuration produced the recorded behaviour |
| Source evidence | Record identifiers, source versions and the relevant retrieved facts | What information was available at the decision point |
| Tool activity | Tool name and version, permitted arguments, attempt number and operation identifier | What was requested and whether an attempt repeated earlier work |
| Timing | Event time, operation start and end, plus completion status | Whether a result arrived before the customer’s answer |
| Outcome | Safe error code, returned record identifier and verified saved status | Whether the action completed and what changed |
| Customer response | The relevant redacted answer or a restricted reference to it | Whether the agent’s claim matches the evidence |
A parent operation identifier connects a tool result to its request. Keep that relationship when work moves into a queue. Label missing events explicitly so an incomplete trace cannot look like a fully observed success.
Redact private details before they enter the trace store. Exclude passwords, access tokens and payment card details. Use task-scoped substitute identifiers for customers, and keep any lookup back to a real person under separate access controls. Free-text fields need particular attention because customers can type private information into an ordinary product question.
The OWASP logging guidance covers sensitive-data exclusion and restricted log access. Give staff access for their diagnostic role, record access and protect traces from unauthorised alteration. Set a retention period for the diagnostic purpose and applicable obligations, including exported copies and backups.
For replay, build a fictional fixture containing ORD-204, product K-40 and policy version 7. Preserve the relevant tool responses without copying a real customer’s address or credentials. This keeps the failure reproducible while limiting what the test exposes.
An AI agents dashboard can show failed tasks and response times, then open the relevant trace. A high success count alone won’t explain why one return claim was wrong. Check whether the dashboard counts accepted requests, completed jobs or confirmed saved actions before using that number.
Verify the correction
Replay the failing task after the correction and compare both the customer’s answer and the saved return state. Use isolated test records and controlled tool responses so replay cannot create a real return or charge.
- Preserve the original fictional inputs and expected result. Task T-81 must never report a created return while job J-9 is pending or failed.
- Apply the focused correction. Make the answer wait for confirmed completion, and route a failed return to the store’s assistance process.
- Replay the pending-then-failed result. Expect an accurate customer message and no saved return. Confirm the trace records the failure and assistance handover.
- Replay a successful result. Expect one saved return with its identifier, followed by a customer answer that matches its status.
- Replay an interrupted response. Look up the original operation before resuming, and confirm the workflow creates no duplicate return.
Keep the corrected case among your repeatable conversation and action checks. Use the same source and configuration versions when comparing results, then record the versions used for the correction.
Passing this case establishes that the chosen failure now produces the expected outcome. It doesn’t establish overall answer quality, faster responses across every task or better search rankings. Use the trace to decide the next correction, and use broader testing to assess the rest of the workflow.
Give your store its own shopping agent
Customers ask in their own words. Your agent finds, compares and checks out from your catalogue, right on your site.