Skip to main content
When an incident is active, four specialized agents handle different stages of the response pipeline. Each agent has a defined scope, tools, and output format. They run sequentially, handing off context between stages.

​
Incident Commander

The Incident Commander coordinates the pipeline. It runs at the start of each incident, between stages, and at closure. What it does:
  • Assigns initial severity and stage based on incoming alerts
  • Decides which specialist agent to run next
  • Performs situational assessments when context is missing (e.g., if a stage was skipped)
  • Infers missing context from available metadata when soft dependencies are absent
  • Triggers the closure stage and generates the incident summary
When it requests HITL approval:
  • The Commander itself does not create approval gates. It routes to the HITL gate after the Diagnosis Specialist’s output.
What it outputs:
  • Stage routing decisions (which agent runs next)
  • Triage classification (severity, affected services, impact summary) in the Triage stage
  • Incident closure summary in the Closure stage
  • Needs-input pauses when hard dependencies are missing (e.g., trying to skip to Resolution without Diagnosis results)
Handoff: After the Commander completes triage, it routes to the Diagnosis Specialist. After closure, the incident is marked resolved and post-closure agents are triggered.

​
Diagnosis Specialist

The Diagnosis Specialist queries your observability tools to identify the root cause of the incident. This is typically the most time-consuming stage. What it does:
  • Queries metrics, logs, and traces via your connected MCP integrations
  • Retrieves relevant runbooks from your knowledge base via vector search
  • Builds a root cause hypothesis grounded in telemetry evidence
  • Produces recommendations with risk classifications
  • Stamps a confidence score per recommendation
What it reads:
  • Telemetry tools (metrics, logs, traces) via MCP
  • Runbooks from .sureops/runbooks/
  • Service topology from your services.yaml contract
  • Prior stage context from detection and triage
When it requests HITL approval:
  • In Guided/Supervised mode, always pauses after producing recommendations — the approval gate lets you review before any action executes.
  • In Self-Driving/autonomous mode with auto-approve enabled (or a recommendation within your configured risk ceiling), lower-risk recommendations can execute without a pause. Two things always force a human interrupt regardless of mode or auto-approve settings: P1/P2 severity incidents, and any fix-PR recommendation. Everything else is eligible to auto-execute only if your organization has opted in.
  • Passes the approved (or auto-approved) recommendations to the Resolution Specialist
What it outputs:
  • Root cause analysis with supporting evidence
  • One or more recommendations, each with: action description, risk level, executor (agent runtime, fix PR, human, escalate, or monitor), and intent (execute-fix, reinvestigate, monitor, or escalate)
  • Confidence score per recommendation
Handoff: After your approval, the HITL node routes approved recommendations to the Resolution Specialist.

​
Re-investigation

If you approve a reinvestigate recommendation, the Diagnosis Specialist runs again with a focus on the area you flagged. Prior rounds remain visible but inactive. The maximum number of re-investigation rounds is configurable (default: 2).

​
Resolution Specialist

The Resolution Specialist executes approved fixes. It runs only after a HITL-approved batch of recommendations. What it does:
  • Executes agent_runtime recommendations (Kubernetes operations, rollbacks, config changes) via your connected MCP integrations
  • Opens a pull request for agent_fix_pr recommendations (code-level changes) — whether it opens as a draft for your review or gets opened already-merged depends on your org’s github_auto_merge setting (silently enabled if you skip providing a GitHub username during onboarding) — except for a forward-diff (LLM-authored) fix, which always opens as a draft requiring manual merge regardless of that setting
  • Records the result of each action (applied, failed, or skipped)
  • Flags when no runtime action was possible (handing off to you if all recommendations require human execution)
What it reads:
  • The approved recommendations from the HITL gate
  • Service deployment metadata from your services.yaml contract (for rollback targets)
  • Execution context from your Kubernetes/ArgoCD MCP integrations
When it requests HITL approval:
  • The Resolution Specialist does not create additional approval gates. The approval gate is pre-resolution (at the Diagnosis Specialist stage).
  • Exception: if a resolution action fails mid-execution or produces no applicable fix, the incident escalates to a human.
What it outputs:
  • Execution log with timestamps and action results
  • A record of which recommendations were applied vs. handed off
  • resolution_no_op flag (triggers human takeover) when nothing could be executed
Handoff: After execution, routes to the Verification Specialist.

​
Executor types

Each recommendation carries an executor that determines how it is handled:

​
Verification Specialist

The Verification Specialist validates that the fix worked. It runs after Resolution completes. What it does:
  • Waits a stabilization period before comparing metrics
  • Queries the same metrics that were anomalous during the incident
  • Compares current values against pre-incident baselines
  • Determines whether recovery is confirmed
What it reads:
  • Resolution results (what was changed, when)
  • Pre-incident metric baselines from the incident’s detection context
  • Current telemetry via your connected MCP integrations
When it requests HITL approval:
  • Does not create approval gates in normal operation
  • If recovery cannot be confirmed, it escalates to the Commander, which presents options (re-run diagnosis, wait longer, or close with caveats)
What it outputs:
  • A metric comparison table: baseline vs. current vs. status (recovered / degraded / inconclusive)
  • A recovery verdict: confirmed, failed, or inconclusive
  • Signals to the Commander that the incident is eligible for closure and Problem Record creation
Handoff: Routes to the Closure stage if recovery is confirmed; back to the Commander for human decision if recovery cannot be confirmed.

​
Stabilization period

Before comparing metrics, the Verification Specialist waits through a brief internal stabilization period. This is a fixed, two-phase wait — it is not severity-scaled and not configurable from any Settings UI. This is a distinct mechanism from the severity-scaled auto-close cooldown you can tune at Settings → Incident Policy (or per environment at the environment’s Incident Policy tab) — that setting controls how long a recovered incident dwells before auto-closing, not the pre-comparison stabilization wait described here.

​
Problem Records on closure

Problem Record creation is driven by your closure policy, not by whether the incident matches a prior pattern. When an incident closes, the Commander evaluates it against your configured severity and closure-reason rules (Settings → Incident Policy) to decide whether a Problem Record is created. Recurrence — whether this incident clusters with prior incidents — is an attribute stamped onto the Problem Record afterward, not a precondition for creating one in the first place. See Post-Closure Agents for what runs once a Problem Record exists.

​
Stage handoff and context preservation

Every stage’s output is persisted to the incident record. When you switch between Self-Driving and Supervised modes, or when you take manual control, the findings from completed stages remain available for the next agent. If you take control mid-diagnosis, the Diagnosis Specialist saves its partial findings. When you release control, the Commander performs a situational assessment and routes to the appropriate stage based on what has been completed. See Supervised vs Self-Driving for details on control handoff.