Prometheus Rule Builder
DevOps & SRE Telemetry Automation

Prometheus Alerting & Recording Rule Generator (PromQL)

Construct PromQL alerting rules, recording rules, alert manager thresholds, labels, and runbook annotations visually.

Prometheus 2.x • K8s Operator CRD • Promtool
Presets:
Rule Configuration & PromQL
Interactive PromQL Builder Assistant
Label Matchers (e.g. {mode="idle"}):
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85Live Assistant
Tier: Range Rate Window
Wait before firing
Prometheus 2.42+ anti-flapping
Warning (P2)
:
:
:
# Prometheus Alerting & Recording Rules Specification
# Generated by ToolsAA (100% Client-Side Engine)
groups:
  - name: infrastructure_node_alerts
    interval: 30s
    rules:
      - alert: HostHighCpuLoad
        expr: 100 - (avg by (instance, job) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        keep_firing_for: 2m
        labels:
          severity: warning
          tier: "infra"
          team: "sre-platform"
        annotations:
          summary: "Host CPU utilization exceeds 85% on {{ $labels.instance }}"
          description: "Node {{ $labels.instance }} (job: {{ $labels.job }}) has had CPU utilization above 85% for 5m. Current load is {{ $value | printf \"%.2f\" }}%."
          runbook_url: "https://runbooks.prometheus-operator.dev/runbooks/general/cpu-high-load/"
          dashboard_url: "https://grafana.internal.net/d/node-exporter-full?var-instance={{ $labels.instance }}"
          action_required: "Investigate noisy-neighbor processes via htop or check autoscaling capacity."
Rule Architecturealert
HostHighCpuLoad

Group: infrastructure_node_alerts | For: 5m

PromQL AnalysisValid
Rate Counter

Range: Windowed | Annotations: 3 vars

Tooling Compatibility100% Native
Prometheus 2.x & K8s

kube-prometheus-stack • Alertmanager • promtool

PromQL AST & Expression Synthesis

Visually construct robust PromQL expressions with metric selectors, range windows, aggregation functions, and comparison thresholds.

100% Browser-Side Privacy

Zero server communications or telemetry. Your internal metric names, cluster topologies, and runbook URLs remain private in your browser.

Promtool & Operator Native

Generates standard Prometheus 2.x rule files, kube-prometheus-stack PrometheusRule CRDs, and promtool test unit specs.

Technical Reference & Architecture Guide

Prometheus Alerting & Recording Rule Generator (PromQL) Developer Guide — Reference Guide

Construct PromQL alerting rules, recording rules, alert manager thresholds, labels, and runbook annotations visually.

Key Parameters & Controls

Rule Type & Identifierstring (alert: / record:)

Specifies whether the YAML block defines an Alerting Rule (`alert: Name`) triggered upon threshold violation or a precomputed Recording Rule (`record: metric:name`).

Default: HighCpuUsage
PromQL Expression (`expr`)string (PromQL syntax)

The metric query evaluated at each evaluation interval to trigger the alert or calculate the recorded time series.

Default: 100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
Pending Duration (`for`)duration string (e.g., 5m, 10m, 1h)

The duration a condition must continuously remain true before switching from `Pending` to `Firing` state, preventing false positives from transient spikes.

Default: 5m
Labels (`labels`)key-value map

Metadata key-value pairs attached to the alert used for Alertmanager notification routing, receiver grouping, silencing, and severity tiering.

Default: severity: warning, team: sre
Annotations (`annotations`)key-value map (supports Go templating)

Human-readable diagnostic information including summary, description, and runbook URLs, supporting Go template interpolations such as `{{ $labels.instance }}` and `{{ $value }}`.

Default: summary: "High CPU load on {{ $labels.instance }}"

Pro Tips & Best Practices

Offload Heavy Queries with Recording Rules

Precompute expensive aggregations or long-window rates (e.g., `rate(http_requests_total[1h])`) using recording rules formatted as `level:metric:operations` to keep dashboard queries and alert evaluation latency minimal.

Format Numeric Values in Annotations with Go Template Filters

Avoid raw unformatted numbers like 0.85231999999 in notifications. Use template filters like `{{ $value | humanizePercentage }}` or `{{ $value | printf "%.2f" }}` for clean, actionable alerts.

Dampen Transient Flapping with Balanced `for` Durations

Set `for: 5m` to `15m` for volatile resource metrics (CPU, disk I/O, JVM GC pauses) to prevent noisy alerts, while keeping synthetic uptime probes (such as Blackbox `probe_success == 0`) to tight windows like `for: 1m`.

Automate Offline Rule Validation with `promtool` in CI/CD

Run `promtool check rules rules.yml` in your deployment pipeline to catch YAML syntax errors, invalid PromQL expressions, and undeclared template variables prior to deploying to production.

Common Gotchas & Troubleshooting

Alert continuously remains in `Pending` state and never transitions to `Firing`

Verify whether the PromQL metric intermittently dips below the threshold during the evaluation window. If the expression evaluates to false even for a single cycle, the pending duration counter resets to zero. Also check that Prometheus `evaluation_interval` is significantly shorter than the rule's `for` duration.

Alert rendering fails with "template: rule: function not defined" error

Prometheus annotations only support built-in template functions (such as `humanize`, `humanize1024`, `humanizeDuration`, `humanizePercentage`, `title`, `toUpper`, `toLower`). Ensure custom shell or string functions are removed and variables are referenced via `$labels.key` or `$value`.

No alerts fire when a monitored microservice or node completely crashes

Standard comparison queries like `http_requests_total > 100` return an empty vector when the target stops exporting metrics. To detect missing instances, configure separate alerting rules using `absent(up{job="api"}) == 1` or check `up{job="api"} == 0`.

Frequently Asked Questions

What is the difference between an Alerting Rule and a Recording Rule in Prometheus?▼

An Alerting Rule evaluates a PromQL expression and dispatches notification events to Alertmanager when conditions hold true over a specified duration. A Recording Rule continuously precomputes complex PromQL queries and writes the evaluated result back into the TSDB as a brand-new time series metric.

How do rule-level labels interact with metric-level labels?▼

Labels defined in the rule's `labels` block take precedence over labels extracted from the scraped metric. If a conflict occurs on the same label key, the value declared in the alert definition overrides the time-series metric value.

Can I use PromQL subqueries within alerting rule expressions?▼

Yes, subqueries such as `rate(http_requests_total[5m])[30m:1m]` are syntactically valid in `expr`, but they incur high CPU and memory overhead during rule evaluations. It is standard practice to compute the inner rate via a Recording Rule first.

How do I write automated unit tests for Prometheus alerting rules?▼

Use `promtool test rules test.yml`. You can define mock time series data, simulated scrapings, and assert whether an alert is expected to be in `pending` or `firing` status at specific time offsets.

What is the recommended Prometheus rule group naming convention?▼

Group rules logically by service, team, or failure domain (e.g., `groups: - name: node_exporter_alerts`). Set appropriate `interval` fields per group if specific evaluations require different polling cadences.

Related Tools & Utilities

All Tools →