Runbooks & Knowledge Base

Runbooks are Markdown files in .sureops/runbooks/. The Diagnosis Specialist and Incident Commander automatically retrieve and read the most relevant runbooks for each incident — you don’t need to explicitly point agents at a runbook. sureops uses vector search to find the best match based on the incident’s symptoms, alerts, and affected services.

How agents use runbooks

During diagnosis, the agent:

Generates a query based on the incident’s alert labels, affected services, and current symptoms
Runs a hybrid search (semantic + keyword) across all runbooks in your knowledge base
Retrieves the top-matching runbooks and includes them as grounding context in its analysis

The incident timeline shows which runbooks were consulted: look for entries like “Consulted runbook: payments.md” in the reasoning section of the Diagnosis Specialist’s output. This lets you verify that the right runbook was retrieved and improve the content if it wasn’t.

Naming conventions

Runbooks live in .sureops/runbooks/ and use the filename as a searchable identifier. Recommended naming patterns:

.sureops/runbooks/
├── payments.md              # One file per service (linked from services.yaml)
├── auth-service.md
├── checkout.md
├── high-error-rate.md       # One file per symptom pattern
├── connection-pool-exhaustion.md
├── deployment-rollback.md   # One file per procedure type
└── database-failover.md

Service-specific runbooks are linked from services.yaml via the runbook field. Pattern-based runbooks are discovered purely through search — no explicit link needed.

Frontmatter schema

Frontmatter fields help the retrieval layer rank and filter results. All are optional, but title and service significantly improve retrieval accuracy.

---
title: "Payments service: degraded throughput"
service: payments                # matches the name field in services.yaml
tags: [latency, retry-storm, connection-pool]
severity: [p1, p2]               # which severities this runbook applies to
last_reviewed: 2026-04-15
---

Field	Type	Description
`title`	string	Required for display. Included in search ranking.
`service`	string	Matches a `name` in `services.yaml`. Boosts ranking when the incident involves this service.
`tags`	list	Keywords for the symptom pattern. Use consistent vocabulary across runbooks.
`severity`	list	Severities this applies to. Used to filter when severity is known.
`last_reviewed`	date	Helps agents note when a runbook may be stale.

Content best practices

Be action-oriented

Agents are better at following specific instructions than at interpreting vague guidance. Prefer:

## Diagnosis steps

Check connection pool utilization: query `redis.connections.used` for the last 15 minutes.
Check the service's error rate: look for `5xx` responses in the access log.
If pool is above 80%, check for long-running queries holding connections.

Over:

## Diagnosis steps

Investigate the issue using your monitoring tools to find the root cause.

Include service-specific error patterns

Agents benefit from knowing what errors your service typically surfaces during failures:

## Known error signatures

- `ETIMEDOUT` in auth-service logs → usually connection pool saturation
- `too many open files` → kernel file descriptor limit
- `dial tcp: connect: connection refused` on port 6379 → Redis is down or unreachable

Link to dashboards

When you reference a dashboard, include the URL directly. Agents can include it in their output so responders can navigate there without searching:

## Key dashboards

- [Payments error rate (Grafana)](https://grafana.acme.com/d/abc/payments-errors)
- [Redis connection pool (Grafana)](https://grafana.acme.com/d/xyz/redis-connections)

Include mitigation steps with commands

For procedures you want the Resolution Specialist to be able to propose:

## Mitigation

### Increase Redis connection pool size

Apply the Helm change:

helm upgrade payments charts/payments —set redis.maxConnections=200 -n prod


This change takes effect within 60 seconds and does not require a pod restart.

The knowledge base

The knowledge base is the collection of all your runbooks, indexed and stored as vector embeddings. Every time you push a change to .sureops/runbooks/, sureops re-indexes the affected files.

Verifying retrieval

After an incident closes, check the Diagnosis Specialist’s timeline entry to confirm the right runbooks were found:

Runbook consulted: the agent found and used a relevant runbook
No matching runbook found: the agent fell back to general knowledge — consider authoring a runbook for this pattern

Keeping runbooks fresh

Stale runbooks can mislead agents. A runbook that describes an old architecture or outdated procedures is worse than no runbook at all. Recommended practices:

Set the last_reviewed frontmatter field and review runbooks that haven’t been touched in 90+ days
After a major incident where the runbook was inaccurate, update it as part of closure
After a successful postmortem, consider whether the postmortem learnings should be folded back into the runbook

Runbook format example

---
title: "Auth service: authentication failures"
service: auth
tags: [auth, 401, token-expiry, redis-session]
severity: [p1, p2]
last_reviewed: 2026-05-10
---

# Auth service: authentication failures

## Symptoms

- Spike in `401 Unauthorized` responses from `/api/v1/auth/token`
- Users reporting login failures or session expiry
- Alert: `auth_error_rate > 5%`

## Known error signatures

- `token validation failed: redis: connection refused` → Redis session store unreachable
- `jwt: token expired` spike with no service restart → NTP skew (check clock drift)
- `too many connections` in auth-db logs → database connection pool exhaustion

## Diagnosis steps

1. Check the auth service error rate over the last 30 minutes.
2. Check Redis connectivity from the auth pods: `kubectl exec -n prod -it auth-pod -- redis-cli -h redis ping`
3. If Redis is unreachable, check Redis pod status: `kubectl get pods -n prod -l app=redis`
4. If Redis is healthy, check for token clock drift: query `auth.jwt.clock_skew` metric.

## Mitigation

### Redis unreachable

Restart the Redis pod if it's crashed:

kubectl rollout restart deployment/redis -n prod

Recovery is typically within 30–60 seconds. Existing sessions may need to re-authenticate.

### Clock drift

Contact your platform team to investigate NTP sync on the auth service pods. This requires a node-level fix and cannot be mitigated by a pod restart.

## Escalation

If auth failures persist after Redis recovery or if the root cause is unclear, escalate to `#team-auth-oncall`.

Getting Started

Onboarding & Setup

Using sureops

Operating Modes & Trust

Integrations

Your Rules (.sureops)

AI Agents & Models

Problem Management

Administration

Self-Hosting

Reference

Runbooks & Knowledge Base

How agents use runbooks

Naming conventions

Frontmatter schema

Content best practices

Be action-oriented

Include service-specific error patterns

Link to dashboards

Include mitigation steps with commands

The knowledge base

Verifying retrieval

Keeping runbooks fresh

Runbook format example

​How agents use runbooks

​Naming conventions

​Frontmatter schema

​Content best practices

​Be action-oriented

​Include service-specific error patterns

​Link to dashboards

​Include mitigation steps with commands

​The knowledge base

​Verifying retrieval

​Keeping runbooks fresh

​Runbook format example

How agents use runbooks

Naming conventions

Frontmatter schema

Content best practices

Be action-oriented

Include service-specific error patterns

Link to dashboards

Include mitigation steps with commands

The knowledge base

Verifying retrieval

Keeping runbooks fresh

Runbook format example