Defining Recording Rules

Overview

Recording Rules are evaluated just like alerting rules but instead of creating notifications, Recording Rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. These pre-calculated queries can be re-used in other rules (including alerting rules) and for dashboards, without the performance impact of repeating the query each time.

Defining Recording Rules

The alert.rules.yml file defines the recording rules along with the alert rules.

The Prometheus Authors provide formal documentation for recording rules.

For example, the following recording rule creates a new metric to pre-calculate the 99th percentile of the duration of each job_type, which can be reused to trigger alerts of different severity (without calculating the quantile twice)

- name: billing_jobs_health
  rules:
  - record: "job_type:billing_jobs_duration_seconds:99p5m"
    expr: histogram_quantile(0.99, sum(rate(jobs_duration_seconds_bucket{service="billing-processing"}[5m])) by (job_type))

  - alert: Billing_Processing_Very_Slow
    expr: "job_type:billing_jobs_duration_seconds:99p5m > 30"
    for: 5m
    labels:
       severity: critical

  - alert: Billing_Processing_Slow
    expr: "job_type:billing_jobs_duration_seconds:99p5m > 15"
    for: 5m
    labels:
       severity: warning