Skip to content

Knowledge base

Agent observability

What to record for every agent run and how to trace a commit back to the agent behind it, for teams running agents in production.

Written by

Updated:

Software Factories 5

Our position. We found that our coding agents got a lot better once we could see what they actually do. Log every prompt, tool call and file change per run, tag each agent commit with the agent, its intent and the prompt behind it, and read the record per workflow, not per call.

The two records answer different questions. The run log explains a failure, and the tag on each change tells you which agent, prompt and intent to fix.

What should you record for every agent run?

The same input can fail differently on each run, because agent work is non-deterministic. The baseline is a log of every prompt, tool call, file change, token count and step duration. That log feeds the retrospectives and log mining described in AgentOps and the compound factory, and the cost controls in inference economics.

A continuous delivery guide sets a minimum record for every agent invocation: timestamp, workflow ID, agent, model, skill, input and output tokens, cache reads and writes, duration, the decision taken and the number of findings. A non-deterministic run cannot be debugged without that record. The guide aggregates per workflow, not per call, because one review command can fan out to four sub-agents and only their sum shows what the command costs. Token fields differ by vendor, so the orchestration layer maps them into one shared schema (MinimumCD.org and DojoConsortium.org - Agentic Architecture Patterns (opens in a new tab), as of March 2026).

One open-source factory shows how cheap that observability foundation can be when the data path is fixed to exactly one shape: agents write to SQLite, readers poll SQLite, and nothing else is permitted. The harness tails the coding agent’s line-delimited JSON output and inserts each event while the phase is still open, so tool calls are visible mid-run instead of batched at the end, landing across seven tables for sessions, phases, events, envelopes, gate results, agent sessions, and the process ids that let a stuck run be found and killed. Because the live view and the full history are the same cursor query at different cadence, the design needs no ingest endpoint, no WebSocket, no backfill, and no separate replay path, and write-ahead logging keeps readers from blocking the running writers. Raw files stay the record and the database is a rebuildable mirror, so losing it costs nothing (Super Simple Software Factory (opens in a new tab)).

How do you trace a change back to its agent?

Without a record of which agent, prompt and intent produced a change, a team cannot learn from an agent failure, audit agent behavior or tighten its constraints. A continuous delivery guide tags every agent commit with the agent identity, the intent description and the prompt or context used, carries the same metadata into deployment records, and reads it in incident retrospectives. The tag also makes the agent metrics measurable: change failure rate is compared between agent and human deployments in the deployment tracker (MinimumCD.org and DojoConsortium.org - Pitfalls and Metrics (opens in a new tab), as of September 2026).

Resources

Documentation

Tools and repositories