Observability Runbook (Queue/Tools/Latency/Embeddings)

Last Updated: 2026-02-28
Scope: Runtime operational diagnosis for the architecture-hardening track.

---

Purpose

Provide a single operational view for the four critical signals required by closure:

• queue backlog/worker health
• tool access denials by reason
• turn latency breakdowns
• embedding usage and failures

This runbook is designed to let on-call diagnose first without reading source code.

---

Primary Data Sources

• Admin endpoint: GET /api/admin/observability
  • returns queue health, denial counters, latency percentiles, embedding totals
• Health endpoints:
  • GET /health/ready
  • GET /health/detailed
• Log events:
  • [ToolExecutor] Tool access denied
  • [ChatFlow] Latency breakdown
  • [ChatFlow] Embedding usage
  • [BackgroundJob][bullmq] Enqueued / failed/completed worker logs

---

Triage Playbook

1) Queue symptoms

• Symptom: queue ok=false or details=worker_not_running.
  • Action:
    • verify Redis connectivity/config (BG_QUEUE_REDIS_URL or REDIS_URL)
    • restart backend worker process
    • confirm GET /api/admin/observability shows workerRunning=true
• Symptom: failed_backlog_high.
  • Action:
    • inspect recent failed jobs in logs
    • verify payload size limit (BG_FILE_JOB_MAX_BYTES)
    • lower enqueue pressure until failures stabilize

2) Tool denial spikes

• Symptom: rising toolDenials.byReason.
  • Action:
    • allowed_protocols_mismatch: validate routed protocol against tool policy
    • explicit_user_selected_required: ensure UI/agent explicitly selected tool
    • self_only_target_mismatch: inspect payload target user and caller user
  • Escalation:
    • if denials block normal user workflows, open policy regression incident

3) Latency regressions

• Symptom: turnLatency.p95TotalTurnMs jumps.
  • Action:
    • inspect latest samples in turnLatency.latest
    • isolate dominant stage (retrievalMs, modelMs, toolMs, etc.)
    • correlate with deployment and traffic changes

4) Embedding failures/cost pressure

• Symptom: high embeddingFailuresInTurn or increasing provider failures.
  • Action:
    • check embedding provider credentials/config
    • verify cache-hit ratio trends
    • reduce noisy embedding callers if failures are transient

---

Release-Gate Evidence Requirements

For architecture-affecting PRs:

• attach latest:
  • npm run test:release:gate
  • npm run env:check-drift
• include snapshot excerpt from GET /api/admin/observability proving:
  • queue health visible
  • denial counters visible
  • latency p50/p95 visible
  • embedding totals visible

---

Escalation Notes

• P0/P1 regressions: block release immediately.
• P2 regressions:
  • if diagnosis remains possible via this runbook and endpoint, classify as non-blocking hardening item
  • if diagnosis is not possible, classify as release-blocking observability failure