> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sureops.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runbooks & Knowledge Base

> How to author runbooks that agents use during incident diagnosis and resolution.

Runbooks are Markdown files in `.sureops/runbooks/`. The Diagnosis Specialist and Incident Commander automatically retrieve and read the most relevant runbooks for each incident — you don't need to explicitly point agents at a runbook. sureops uses vector search to find the best match based on the incident's symptoms, alerts, and affected services.

## How agents use runbooks

During diagnosis, the agent:

1. Generates a query based on the incident's alert labels, affected services, and current symptoms
2. Runs a hybrid search (semantic + keyword) across all runbooks in your knowledge base
3. Retrieves the top-matching runbooks and includes them as grounding context in its analysis

The incident timeline shows which runbooks were consulted: look for entries like "Consulted runbook: payments.md" in the reasoning section of the Diagnosis Specialist's output. This lets you verify that the right runbook was retrieved and improve the content if it wasn't.

***

## Naming conventions

Runbooks live in `.sureops/runbooks/` and use the filename as a searchable identifier. Recommended naming patterns:

```
.sureops/runbooks/
├── payments.md              # One file per service (linked from services.yaml)
├── auth-service.md
├── checkout.md
├── high-error-rate.md       # One file per symptom pattern
├── connection-pool-exhaustion.md
├── deployment-rollback.md   # One file per procedure type
└── database-failover.md
```

Service-specific runbooks are linked from `services.yaml` via the `runbook` field. Pattern-based runbooks are discovered purely through search — no explicit link needed.

***

## Frontmatter schema

Frontmatter fields help the retrieval layer rank and filter results. All are optional, but `title` and `service` significantly improve retrieval accuracy.

```markdown theme={null}
---
title: "Payments service: degraded throughput"
service: payments                # matches the name field in services.yaml
tags: [latency, retry-storm, connection-pool]
severity: [p1, p2]               # which severities this runbook applies to
last_reviewed: 2026-04-15
---
```

| Field           | Type   | Description                                                                                  |
| --------------- | ------ | -------------------------------------------------------------------------------------------- |
| `title`         | string | Required for display. Included in search ranking.                                            |
| `service`       | string | Matches a `name` in `services.yaml`. Boosts ranking when the incident involves this service. |
| `tags`          | list   | Keywords for the symptom pattern. Use consistent vocabulary across runbooks.                 |
| `severity`      | list   | Severities this applies to. Used to filter when severity is known.                           |
| `last_reviewed` | date   | Helps agents note when a runbook may be stale.                                               |

***

## Content best practices

### Be action-oriented

Agents are better at following specific instructions than at interpreting vague guidance. Prefer:

```markdown theme={null}
## Diagnosis steps

1. Check connection pool utilization: query `redis.connections.used` for the last 15 minutes.
2. Check the service's error rate: look for `5xx` responses in the access log.
3. If pool is above 80%, check for long-running queries holding connections.
```

Over:

```markdown theme={null}
## Diagnosis steps

Investigate the issue using your monitoring tools to find the root cause.
```

### Include service-specific error patterns

Agents benefit from knowing what errors your service typically surfaces during failures:

```markdown theme={null}
## Known error signatures

- `ETIMEDOUT` in auth-service logs → usually connection pool saturation
- `too many open files` → kernel file descriptor limit
- `dial tcp: connect: connection refused` on port 6379 → Redis is down or unreachable
```

### Link to dashboards

When you reference a dashboard, include the URL directly. Agents can include it in their output so responders can navigate there without searching:

```markdown theme={null}
## Key dashboards

- [Payments error rate (Grafana)](https://grafana.acme.com/d/abc/payments-errors)
- [Redis connection pool (Grafana)](https://grafana.acme.com/d/xyz/redis-connections)
```

### Include mitigation steps with commands

For procedures you want the Resolution Specialist to be able to propose:

```markdown theme={null}
## Mitigation

### Increase Redis connection pool size

Apply the Helm change:
```

helm upgrade payments charts/payments --set redis.maxConnections=200 -n prod

```

This change takes effect within 60 seconds and does not require a pod restart.
```

***

## The knowledge base

The knowledge base is the collection of all your runbooks, indexed and stored as vector embeddings. Every time you push a change to `.sureops/runbooks/`, sureops re-indexes the affected files.

### Verifying retrieval

After an incident closes, check the Diagnosis Specialist's timeline entry to confirm the right runbooks were found:

* **Runbook consulted**: the agent found and used a relevant runbook
* **No matching runbook found**: the agent fell back to general knowledge — consider authoring a runbook for this pattern

### Keeping runbooks fresh

Stale runbooks can mislead agents. A runbook that describes an old architecture or outdated procedures is worse than no runbook at all.

Recommended practices:

* Set the `last_reviewed` frontmatter field and review runbooks that haven't been touched in 90+ days
* After a major incident where the runbook was inaccurate, update it as part of closure
* After a successful postmortem, consider whether the postmortem learnings should be folded back into the runbook

***

## Runbook format example

```markdown theme={null}
---
title: "Auth service: authentication failures"
service: auth
tags: [auth, 401, token-expiry, redis-session]
severity: [p1, p2]
last_reviewed: 2026-05-10
---

# Auth service: authentication failures

## Symptoms

- Spike in `401 Unauthorized` responses from `/api/v1/auth/token`
- Users reporting login failures or session expiry
- Alert: `auth_error_rate > 5%`

## Known error signatures

- `token validation failed: redis: connection refused` → Redis session store unreachable
- `jwt: token expired` spike with no service restart → NTP skew (check clock drift)
- `too many connections` in auth-db logs → database connection pool exhaustion

## Diagnosis steps

1. Check the auth service error rate over the last 30 minutes.
2. Check Redis connectivity from the auth pods: `kubectl exec -n prod -it auth-pod -- redis-cli -h redis ping`
3. If Redis is unreachable, check Redis pod status: `kubectl get pods -n prod -l app=redis`
4. If Redis is healthy, check for token clock drift: query `auth.jwt.clock_skew` metric.

## Mitigation

### Redis unreachable

Restart the Redis pod if it's crashed:
```

kubectl rollout restart deployment/redis -n prod

```

Recovery is typically within 30–60 seconds. Existing sessions may need to re-authenticate.

### Clock drift

Contact your platform team to investigate NTP sync on the auth service pods. This requires a node-level fix and cannot be mitigated by a pod restart.

## Escalation

If auth failures persist after Redis recovery or if the root cause is unclear, escalate to `#team-auth-oncall`.
```