Skip to main content
When an incident is active, four specialized agents handle different stages of the response pipeline. Each agent has a defined scope, tools, and output format. They run sequentially, handing off context between stages.

​
Incident Commander

The Incident Commander coordinates the pipeline. It runs at the start of each incident, between stages, and at closure. What it does:
  • Assigns initial severity and stage based on incoming alerts
  • Decides which specialist agent to run next
  • Performs situational assessments when context is missing (e.g., if a stage was skipped)
  • Infers missing context from available metadata when soft dependencies are absent
  • Triggers the closure stage and generates the incident summary
When it requests HITL approval:
  • The Commander itself does not create approval gates. It routes to the HITL gate after the Diagnosis Specialist’s output.
What it outputs:
  • Stage routing decisions (which agent runs next)
  • Triage classification (severity, affected services, impact summary) in the Triage stage
  • Incident closure summary in the Closure stage
  • Needs-input pauses when hard dependencies are missing (e.g., trying to skip to Resolution without Diagnosis results)
Handoff: After the Commander completes triage, it routes to the Diagnosis Specialist. After closure, the incident is marked resolved and post-closure agents are triggered.

​
Diagnosis Specialist

The Diagnosis Specialist queries your observability tools to identify the root cause of the incident. This is typically the most time-consuming stage. What it does:
  • Queries metrics, logs, and traces via your connected MCP integrations
  • Retrieves relevant runbooks from your knowledge base via vector search
  • Builds a root cause hypothesis grounded in telemetry evidence
  • Produces recommendations with risk classifications
  • Stamps a confidence score per recommendation
What it reads:
  • Telemetry tools (metrics, logs, traces) via MCP
  • Runbooks from .sureops/runbooks/
  • Service topology from your services.yaml contract
  • Prior stage context from detection and triage
When it requests HITL approval:
  • Always pauses after producing recommendations — the approval gate lets you review before any action executes
  • Passes the approved recommendations to the Resolution Agent
What it outputs:
  • Root cause analysis with supporting evidence
  • One or more recommendations, each with: action description, risk level, executor (agent runtime, fix PR, or human), and intent (execute-fix, reinvestigate, monitor, or escalate)
  • Confidence score per recommendation
Handoff: After your approval, the HITL node routes approved recommendations to the Resolution Agent.

​
Re-investigation

If you approve a reinvestigate recommendation, the Diagnosis Specialist runs again with a focus on the area you flagged. Prior rounds remain visible but inactive. The maximum number of re-investigation rounds is configurable (default: 2).

​
Resolution Agent

The Resolution Agent executes approved fixes. It runs only after a HITL-approved batch of recommendations. What it does:
  • Executes agent_runtime recommendations (Kubernetes operations, rollbacks, config changes) via your connected MCP integrations
  • Opens a draft pull request for agent_fix_pr recommendations (code-level changes)
  • Records the result of each action (applied, failed, or skipped)
  • Flags when no runtime action was possible (handing off to you if all recommendations require human execution)
What it reads:
  • The approved recommendations from the HITL gate
  • Service deployment metadata from your services.yaml contract (for rollback targets)
  • Execution context from your Kubernetes/ArgoCD MCP integrations
When it requests HITL approval:
  • The Resolution Agent does not create additional approval gates. The approval gate is pre-resolution (at the Diagnosis Specialist stage).
  • Exception: if a resolution action fails mid-execution or produces no applicable fix, the incident escalates to a human.
What it outputs:
  • Execution log with timestamps and action results
  • A record of which recommendations were applied vs. handed off
  • resolution_no_op flag (triggers human takeover) when nothing could be executed
Handoff: After execution, routes to the Verification Agent.

​
Executor types

Each recommendation carries an executor that determines how it is handled:

​
Verification Agent

The Verification Agent validates that the fix worked. It runs after Resolution completes. What it does:
  • Waits a configurable stabilization period (default: configurable per environment)
  • Queries the same metrics that were anomalous during the incident
  • Compares current values against pre-incident baselines
  • Determines whether recovery is confirmed
What it reads:
  • Resolution results (what was changed, when)
  • Pre-incident metric baselines from the incident’s detection context
  • Current telemetry via your connected MCP integrations
When it requests HITL approval:
  • Does not create approval gates in normal operation
  • If recovery cannot be confirmed, it escalates to the Commander, which presents options (re-run diagnosis, wait longer, or close with caveats)
What it outputs:
  • A metric comparison table: baseline vs. current vs. status (recovered / degraded / inconclusive)
  • A recovery verdict: confirmed, failed, or inconclusive
  • Trigger for Problem Record creation if the same pattern has been seen before
Handoff: Routes to the Closure stage if recovery is confirmed; back to the Commander for human decision if recovery cannot be confirmed.

​
Stage handoff and context preservation

Every stage’s output is persisted to the incident record. When you switch between Self-Driving and Supervised modes, or when you take manual control, the findings from completed stages remain available for the next agent. If you take control mid-diagnosis, the Diagnosis Specialist saves its partial findings. When you release control, the Commander performs a situational assessment and routes to the appropriate stage based on what has been completed. See Supervised vs Self-Driving for details on control handoff.