> ## Documentation Index > Fetch the complete documentation index at: https://docs.sureops.ai/llms.txt > Use this file to discover all available pages before exploring further. # Incident Pipeline Agents > The four agents that run during an active incident — Commander, Diagnosis Specialist, Resolution Agent, and Verification Agent. When an incident is active, four specialized agents handle different stages of the response pipeline. Each agent has a defined scope, tools, and output format. They run sequentially, handing off context between stages. ``` Detection → Triage → Diagnosis → [HITL gate] → Resolution → Verification → Closure ↑ ↑ ↑ ↑ ↑ ↑ Commander Commander Diagnosis Resolution Verification Commander ``` *** ## Incident Commander The Incident Commander coordinates the pipeline. It runs at the start of each incident, between stages, and at closure. **What it does:** * Assigns initial severity and stage based on incoming alerts * Decides which specialist agent to run next * Performs situational assessments when context is missing (e.g., if a stage was skipped) * Infers missing context from available metadata when soft dependencies are absent * Triggers the closure stage and generates the incident summary **When it requests HITL approval:** * The Commander itself does not create approval gates. It routes to the HITL gate after the Diagnosis Specialist's output. **What it outputs:** * Stage routing decisions (which agent runs next) * Triage classification (severity, affected services, impact summary) in the Triage stage * Incident closure summary in the Closure stage * Needs-input pauses when hard dependencies are missing (e.g., trying to skip to Resolution without Diagnosis results) **Handoff:** After the Commander completes triage, it routes to the Diagnosis Specialist. After closure, the incident is marked resolved and post-closure agents are triggered. *** ## Diagnosis Specialist The Diagnosis Specialist queries your observability tools to identify the root cause of the incident. This is typically the most time-consuming stage. **What it does:** * Queries metrics, logs, and traces via your connected MCP integrations * Retrieves relevant runbooks from your knowledge base via vector search * Builds a root cause hypothesis grounded in telemetry evidence * Produces recommendations with risk classifications * Stamps a confidence score per recommendation **What it reads:** * Telemetry tools (metrics, logs, traces) via MCP * Runbooks from `.sureops/runbooks/` * Service topology from your `services.yaml` contract * Prior stage context from detection and triage **When it requests HITL approval:** * Always pauses after producing recommendations — the approval gate lets you review before any action executes * Passes the approved recommendations to the Resolution Agent **What it outputs:** * Root cause analysis with supporting evidence * One or more recommendations, each with: action description, risk level, executor (agent runtime, fix PR, or human), and intent (execute-fix, reinvestigate, monitor, or escalate) * Confidence score per recommendation **Handoff:** After your approval, the HITL node routes approved recommendations to the Resolution Agent. ### Re-investigation If you approve a `reinvestigate` recommendation, the Diagnosis Specialist runs again with a focus on the area you flagged. Prior rounds remain visible but inactive. The maximum number of re-investigation rounds is configurable (default: 2). *** ## Resolution Agent The Resolution Agent executes approved fixes. It runs only after a HITL-approved batch of recommendations. **What it does:** * Executes `agent_runtime` recommendations (Kubernetes operations, rollbacks, config changes) via your connected MCP integrations * Opens a draft pull request for `agent_fix_pr` recommendations (code-level changes) * Records the result of each action (applied, failed, or skipped) * Flags when no runtime action was possible (handing off to you if all recommendations require human execution) **What it reads:** * The approved recommendations from the HITL gate * Service deployment metadata from your `services.yaml` contract (for rollback targets) * Execution context from your Kubernetes/ArgoCD MCP integrations **When it requests HITL approval:** * The Resolution Agent does not create additional approval gates. The approval gate is pre-resolution (at the Diagnosis Specialist stage). * Exception: if a resolution action fails mid-execution or produces no applicable fix, the incident escalates to a human. **What it outputs:** * Execution log with timestamps and action results * A record of which recommendations were applied vs. handed off * `resolution_no_op` flag (triggers human takeover) when nothing could be executed **Handoff:** After execution, routes to the Verification Agent. ### Executor types Each recommendation carries an executor that determines how it is handled: | Executor | Meaning | Handled by | | --------------- | ---------------------------------- | ----------------------------- | | `agent_runtime` | Direct execution via runtime tools | Resolution Agent runs it | | `agent_fix_pr` | Code change required | Fix-PR Agent opens a draft PR | | `human` | Human action required | Handed off with instructions | | `escalate` | Route to another team | Triggers human takeover | *** ## Verification Agent The Verification Agent validates that the fix worked. It runs after Resolution completes. **What it does:** * Waits a configurable stabilization period (default: configurable per environment) * Queries the same metrics that were anomalous during the incident * Compares current values against pre-incident baselines * Determines whether recovery is confirmed **What it reads:** * Resolution results (what was changed, when) * Pre-incident metric baselines from the incident's detection context * Current telemetry via your connected MCP integrations **When it requests HITL approval:** * Does not create approval gates in normal operation * If recovery cannot be confirmed, it escalates to the Commander, which presents options (re-run diagnosis, wait longer, or close with caveats) **What it outputs:** * A metric comparison table: baseline vs. current vs. status (recovered / degraded / inconclusive) * A recovery verdict: confirmed, failed, or inconclusive * Trigger for Problem Record creation if the same pattern has been seen before **Handoff:** Routes to the Closure stage if recovery is confirmed; back to the Commander for human decision if recovery cannot be confirmed. *** ## Stage handoff and context preservation Every stage's output is persisted to the incident record. When you switch between Self-Driving and Supervised modes, or when you take manual control, the findings from completed stages remain available for the next agent. If you take control mid-diagnosis, the Diagnosis Specialist saves its partial findings. When you release control, the Commander performs a situational assessment and routes to the appropriate stage based on what has been completed. See [Supervised vs Self-Driving](/next/operating-modes/supervised-vs-self-driving) for details on control handoff.