Skip to main content
When an incident closes and a Problem Record is created for it, a second pipeline runs asynchronously to build out the RCA, critique the response, propose improvements, and (where opted in) draft a code fix. This pipeline runs without blocking the incident response workflow — it produces artifacts you review on your own schedule. Problem Record creation itself is driven by your closure policy — configured per severity and closure reason at Settings → Incident Policy — not by whether the incident matches a pattern seen before. Recurrence is stamped onto the resulting Problem Record (via clustering against prior incidents) as an attribute, not a precondition for creating it. See Incident Pipeline Agents for how a Problem Record becomes eligible at closure.

​
Problem Commander

The Problem Commander is the orchestrator for this pipeline — it’s the agent shown in Settings → AI Agents and the one you configure a model for. Internally it runs a fixed sequence of stages, sequentially, each handing its output to the next:
What triggers it: a Problem Record is created for a closed incident (per your closure policy), or a manual regenerate is requested from the Problem Record page.

​
Deep RCA Investigator

The deep_rca stage produces the root cause analysis. What it does:
  • Correlates the incident with prior related incidents
  • Produces a root cause analysis (RCA) grounded in the incident’s telemetry and timeline
  • Identifies patterns in timing, affected services, and contributing factors
Where results appear:
  • Problem Records page: the Problem Record’s RCA content
  • Feeds every downstream stage (critique, recommendations, fix PR, runbook)
  • If your knowledge base repository is configured, the deep_rca stage also opens a pull request adding the postmortem document to .sureops/postmortems/ — a separate PR from the runbook PR described below, gated by the org-wide automation kill switch (Settings → Knowledge Base → Advanced). There’s also a second, .sureops/policy.yaml-level gate for this one: a postmortem_pr.auto_pr setting (default: true) that can be turned off per KB repo to suppress the postmortem PR without touching the kill switch.
HITL approval: Whether the postmortem PR waits as a draft for your review or gets auto-merged depends on your org’s github_auto_merge setting. That setting is silently enabled if you skip providing a GitHub username during onboarding (see Guided Onboarding Journey) — with a username on file, it stays off and the PR waits as a draft for your team’s normal review and merge.

​
Incident Response Critic

The critic stage audits how the response itself performed — not the underlying system fault. What it does:
  • Produces a customer-facing critique of your detection and response (what your monitoring and response stages caught, and what they missed) — this lands in the Problem Record as the response critique

​
Improvement Recommender

The recommender stage turns the critique into concrete, customer-actionable output. What it does:
  • Produces process improvements you can make to your own incident response process
  • Proposes monitor definitions (Grafana today — Datadog is planned but not yet connectable) that would have caught this incident automatically — these land as staged, reviewable monitor proposals. Pushing a proposed monitor live to your provider is a separate approval step and is gated behind the proposed-monitors feature for your plan.
Where results appear:
  • Problem Records page: process improvements and proposed monitors on the Problem Record

​
Fix-PR Agent

The fix_pr stage attempts to produce a code-level fix for the root cause identified during diagnosis. What triggers it: the RCA from the deep_rca stage, for the same Problem Record — it’s a node in the Problem Commander pipeline, not a standalone handoff from the Resolution Specialist. Before any of that, your org must be on at least a Starter-tier plan for the fix-PR-generation feature itself — this gate isn’t bypassable, not even by a manual trigger. Above that floor, it is off by default: on an auto trigger, it only opens a PR when both of the following are true — the affected service has source_fix_pr.auto_fix_pr: true in its .sureops/policy.yaml entry, and the org-wide automation kill switch isn’t paused (Settings → Knowledge Base → Advanced). Either one being false blocks the auto-triggered PR. You can also trigger it manually from a Problem Record — that bypasses the auto_fix_pr and kill-switch checks, but it still respects the plan-tier gate. .sureops/policy.yaml fields for this agent all live nested under a source_fix_pr block per service, and that block itself can be set at four layers — most specific wins: per_service.<name>.environments.<env>.source_fix_pr, then per_service.<name>.source_fix_pr, then environments.<env>.source_fix_pr, then defaults.source_fix_pr. This lets you set an org-wide baseline once and override only what changes per service or per environment. What it does, depending on your service’s resolved source_fix_pr policy:
  • Revert mode (source_fix_pr.allow_revert: true): when the RCA attributes the incident to a specific commit, opens a draft PR carrying the deterministic inverse diff of that commit — equivalent to git revert, computed by the source-code provider with no LLM involved.
  • Forward-diff mode (source_fix_pr.allow_forward_diff: true): for incidents with no single attributable commit (e.g. a resource limit that needs raising), an LLM authors a structured forward edit, which still goes through a deterministic grounding step before it’s spliced into the diff.
  • All three gates are opt-in per service and disabled by default (source_fix_pr.auto_fix_pr: false, source_fix_pr.allow_revert: false, source_fix_pr.allow_forward_diff: false).
  • Requests review from reviewers configured in that service’s resolved source_fix_pr.reviewers.
Where results appear:
  • A pull request in your GitHub repository (draft or already merged, depending on your github_auto_merge setting — see HITL approval below)
  • The incident and Problem Record gain a “Fix PR” link pointing to the PR
HITL approval: For revert-mode PRs, whether it waits as a draft for your review or gets auto-merged depends on your org’s github_auto_merge setting (same setting, same silent-enable-by-skipping-onboarding behavior described for the postmortem PR above). Forward-diff mode is the one exception: an LLM-authored forward edit always opens as a draft and always requires your manual review and merge — the agent never auto-merges it, regardless of your github_auto_merge setting. In every mode, the agent never pushes directly to a protected branch.

​
Runbook drafting

The final runbook stage assembles a runbook from everything the earlier stages produced (RCA, critique, action items, timeline) and opens a separate pull request — distinct from the postmortem PR described under Deep RCA Investigator above — adding it to your knowledge base repository under .sureops/runbooks/. Like the postmortem PR, this one is gated by both the org-wide automation kill switch (Settings → Knowledge Base → Advanced) and the same postmortem_pr.auto_pr policy.yaml setting described above — no separate per-service opt-in beyond that shared flag. HITL approval: Like the postmortem PR, whether the runbook PR waits as a draft for your review or gets auto-merged depends on your org’s github_auto_merge setting — silently enabled if you skip providing a GitHub username during onboarding (see Guided Onboarding Journey).

​
Pre-mortem Agent (roadmap)

Pre-mortems — automated risk analysis on a pull request before it ships — are not yet available; they’re on the roadmap. See Pricing & Tiers for current status. When shipped, the design is: a GitHub webhook fires on a pull request being opened or updated (synchronize), and the agent reviews the diff against service history and dependency blast radius to produce a pre-mortem summary — likelihood of incident, risk areas, recommended monitoring during rollout — posted as a PR comment. It would be advisory only and would never block a deployment.

​
Configuring post-closure behavior

Post-closure agent behavior is configured in:
  • Settings → AI Agents: Problem Commander’s model
  • .sureops/policy.yaml (source_fix_pr block — settable at defaults, per-environment, and per-service layers): Fix-PR auto-open, draft mode, revert/forward-diff opt-in, and reviewer routing
  • Settings → Code fixes: org-scoped repo mappings and auto-open behavior for code-fix PRs
  • Settings → Incident Policy: which closed incidents produce a Problem Record in the first place
  • Settings → Knowledge Base → Advanced: the org-wide automation kill switch that gates the fix-PR, postmortem, and runbook PRs