What observability means for AI agents
For traditional software, observability means knowing whether a service is up and whether requests succeed or fail. For AI agents, it means considerably more: knowing what the agent read from your systems, what reasoning it applied to that data, what action it proposed, who approved it, and whether the write succeeded.
Without this level of trace, you cannot diagnose incorrect behaviour, respond to incidents, or demonstrate compliance. For SMB and mid-market teams deploying agents across Salesforce, Zendesk, Slack, and QuickBooks, observability is what makes production deployments safe.
Key observability signals
| Signal | What it captures | Why it matters |
|---|---|---|
| Run trace | Full record of what the agent read, reasoned, and proposed per run | Root cause analysis and audit compliance |
| Decision log | Each reasoning step with inputs and outputs | Understand why an agent proposed a specific action |
| Approval latency | Time between proposed action and human approval | Identify bottlenecks in approval workflows |
| Write success rate | Percentage of proposed writes that execute without error | Detect stack connectivity issues early |
| Cost per run | API and compute cost per agent run | Track against spend caps, detect runaway usage |
| Error rate | Failed runs as percentage of total runs | Early warning for permission changes or stack API issues |
| Pending approval queue | Actions waiting for human review | Prevent workflow backlogs from blocking automation |
Monitoring setup
Set up these monitoring views before deploying agents to production.
Run dashboard
- Total runs per agent per day
- Success rate per agent
- Average cost per run vs spend cap
- Pending approvals queue depth
Decision trace view
- Searchable log of all agent runs
- Per-run view: inputs read, reasoning steps, actions proposed
- Approval history per proposed action
- Write outcomes with timestamps and stack confirmation
Stack health view
- OAuth token status per connected stack
- API error rates per stack (Salesforce, Zendesk, Slack, QuickBooks)
- Last successful run per agent
Alerting rules
Configure these alerts before going to production. Tune thresholds to match your team's normal operating patterns after the first two weeks.
Error rate spike
Alert when error rate exceeds 20% in a 1-hour window. Common causes: stack API change, revoked OAuth token, schema change in connected tool.
Cost cap approach
Alert when an agent reaches 80% of its daily spend cap. Gives you time to review before the cap triggers a pause.
Stale approval queue
Alert when any approval action has been pending for more than 4 hours. Prevents workflow backlogs from silently blocking automation.
Credential expiry
Alert 7 days before an OAuth token expires. Gives time for the credential owner to reauthorise without agent downtime.
High rejection rate
Alert when more than 30% of proposed actions are rejected by approvers in a day. Signals the agent instructions need tuning.
Zero-run agent
Alert when an agent that normally runs daily has not run in 48 hours. Detects silent failures from schedule or trigger issues.
Frequently asked questions
What is the difference between AI agent observability and traditional application monitoring?
Traditional monitoring checks whether a system is up and whether requests succeed. Agent observability goes further: it traces the reasoning chain (what the agent read, what it inferred, what it proposed), logs every human approval decision, and surfaces cost and quality metrics specific to agentic workflows.
Do I need a dedicated observability platform for AI agents?
Not necessarily. If you are deploying agents through Pinksheep, observability is built in: every run produces a trace, every approval is logged, and spend caps trigger alerts automatically. You only need a separate observability platform if you are building custom agent infrastructure.
How do we know if an agent is behaving incorrectly before it causes problems?
Set up anomaly alerts on error rate, approval rejection rate, and cost per run. A spike in any of these is an early signal of a problem. Review the run trace for the affected period to identify the root cause before it escalates.
How long should we retain agent run logs?
Retain run logs for at least 90 days for operational troubleshooting. For agents that touch finance, HR, or compliance-sensitive systems, retain logs for 12 months minimum to support audit requirements.
Can non-technical team members review agent behaviour?
Yes. The run trace and approval log should be accessible to the business owner of each workflow, not just the technical team. Non-technical reviewers need to see what the agent proposed and why, without reading raw logs.