Skip to main content
Proactive detection watches your telemetry continuously and opens low-confidence incidents when it spots statistical anomalies — often 5–30 minutes before your alert thresholds would fire. By the time your real alert arrives, sureops already has context from the early detection.

​
How it works

Every minute, proactive detection:
  1. Polls your observability data — checks the last 5 minutes of metrics per service
  2. Compares against a 7-day baseline — using the same time-of-day window (e.g., “Mondays 9–10 AM baseline”) to account for traffic patterns
  3. Flags anomalies — signals where the current metric deviates more than 2 standard deviations (z-score > 2.0) from the baseline
  4. Computes confidence — factors in spike intensity, whether multiple metrics are spiking together, metric volatility, time-of-day context, and false-alarm history for that service
  5. Opens a proactive incident with confidence 35–60% if all conditions are met
A proactive incident is distinct from a standard alert-triggered incident. It is marked clearly in the Incidents list and on the Command Center.

​
Metrics monitored

By default, proactive detection watches: Additional metrics (goroutine exhaustion, memory usage) can be enabled per-service in Settings.

​
Confidence scoring

Proactive incidents are opened with confidence in the 35–60% range. The score reflects how certain the system is that this is a real problem:
  • 35–45% — weak signal; single metric, moderate deviation, noisy service history
  • 45–55% — moderate signal; multiple metrics correlating, or extreme deviation on one
  • 55–60% — strong signal; multiple metrics spiking together with extreme deviation
High-confidence proactive incidents (55%+) are more likely to be real. Low-confidence ones (35–40%) should be treated as early warnings that may self-resolve.

​
Responding to a proactive incident

When a proactive incident appears, you have three options: Confirm — you believe this is a real problem. The incident advances to triage and the full agent pipeline kicks in. The system records a positive signal: similar patterns for this service get a confidence boost next time. Dismiss — you believe this is a false alarm. The incident is cancelled. The system records a negative signal: similar patterns for this service get a confidence reduction next time. Wait — if you do not act within 10 minutes and no real alert arrives, the incident is automatically dismissed with reason self_healing. This is a weak negative signal — confidence for this pattern is reduced slightly.

​
What happens when a real alert follows

If a real alert fires for the same service while a proactive incident is still pending:
  1. A standard incident is opened from the alert
  2. The proactive incident is automatically promoted to triage — its early analysis is not discarded
  3. Both incidents are linked via related_incidents so you can see “this incident was predicted 10 minutes ago”
  4. The diagnosis agent continues with the full context from the early detection
  5. The early detection is recorded as correct — confidence for this pattern increases

​
Setting up proactive detection

Proactive detection requires that your OTel collector is forwarding metrics to sureops. Span and log forwarding alone is not sufficient. Enable or disable:
  1. Go to Settings → Anomaly Detection → Proactive Detection
  2. Toggle Enable Proactive Detection on or off
Tune sensitivity: Suppress for noisy services: Mark naturally noisy services (e.g., real-time analytics pipelines with high metric variance) in your .sureops/services.yaml:
Proactive detection skips services marked this way.

​
Seeing proactive incidents in the Incidents list

Proactive incidents appear in the Incidents list with a distinct badge. To hide them when they are too noisy:
  1. Go to Incidents
  2. Toggle Hide proactive detections in the filter bar
Hiding them does not cancel them — they are still created and tracked.

​
How detection improves over time

Every time you confirm or dismiss a proactive incident, the system updates its confidence model for that service and pattern: Over time, the false-alarm rate drops as detection learns the normal behavior of your specific environment.

​
Baseline startup period

New services have no 7-day baseline. For the first week after a service appears in your landscape, proactive detection uses a shorter baseline (last 24 hours) and applies more conservative thresholds. After 7 days of data, detection graduates to the full baseline window.

​
Troubleshooting

Too many false alarms:
  • Raise the z-score threshold from 2.0 to 2.5
  • Reduce the false-positive budget so detection auto-pauses sooner
  • Mark noisy services as is_monitoring_candidate: false
  • Keep dismissing false alarms — the model learns
Missing real issues:
  • Lower the z-score threshold (more sensitive, but more false alarms)
  • Verify your OTel collector is forwarding metrics (not just spans and logs) to sureops
  • Check that the service has at least 7 days of baseline data in your landscape
Service appears in a proactive incident but not in my Landscape:
  • Proactive detection only monitors services with a known otel_service_name in the landscape
  • Add the service to .sureops/services.yaml or wait for the next telemetry sync to discover it
Proactive incidents keep auto-dismissing:
  • The pattern may be a normal fluctuation for this service. Disable detection for it via is_monitoring_candidate: false, or increase the auto-dismiss timeout in Settings.