Defining Alerts

Alerting for Metrics - Prometheus and Alertmanager

Overview

There are 2 parts to defining your Alert configurations for metrics:

• Defining the metrics and values on which to alert, which is a Prometheus configuration.
• Defining "routes" to incident management system like Ocean, which is an Alertmanager configuration.

You update both files using HTTP PUT requests to the Sauron API Server.

When you perform an update, the API server triggers immediate uptake of the modified file by the affected service (Prometheus or Alertmanager).

Defining Metrics Alert Rules

The alert.rules.yml file defines the alert conditions, but does not specify alert routing. The Prometheus Authors provide formal documentation for alert rules.

The alerting rules are specified using YAML. As always, YAML file are bound by rigid formatting rules. Example:

groups:
- name: example
  rules:
  - alert: HighErrorRate
    expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
    for: 10m
    labels:
      severity: page
    annotations:
      summary: High request latency

A text file containing this definition can be uploaded to Prometheus via the Sauron API Server like this:

curl -u sauron:mypassword \
-i -T alert.rules.yml \
https://api.handu-phx.handu.developers.oracledx.com/prometheus/rules/alert.rules.yml

Multiple rules files are allowed.

You can verify the changes through the Prometheus UI: https://prometheus.handu-phx.handu.developers.oracledx.com/alerts.

Defining Metrics Alert Routes

The alertmanager.yml file defines alert routing and other higher-level functionality, but does not define the alert conditions themselves. The Prometheus Authors provide formal documentation for Alertmanager configuration.

Here is an example of Alertmanager configuration:

route:
  receiver: Test
receivers:
  - name: Test

This file, called folder/alertmanager.yml in the example above, can be uploaded to Alertmanager via the Sauron API Server like:

curl -u sauron:mypassword \
-i -T folder/alertmanager.yml \
https://api.handu-phx.handu.developers.oracledx.com/#/Alertmanager/putAlertmanagerConfig

Defining Ocean Integration for Metrics Alerts

route:
  receiver: Testing
  group_by: ['...']
receivers:
  - name: Testing
    webhook_configs:
    - http_config:
        authentication:
          credentials: <bearer_token>
      url: https://oceanclient.ocs.oraclecloud.com/api/v1/webhooks/json/prometheus-alertmanager

To obtain your Ocean webhook bearer_token, please follow instructions in https://ocean.ocs.oraclecloud.com/help

Defining Email Integration for Alertmanager Alerts

Here is a quick walkthrough...

Pre-requisite 1: The following steps require access to an OCI tenancy. Please contact your team or business unit if you don't have your own OCI tenancy.

1. Generate SMTP Credential
   1. Login OCI console
   2. Click "Profile" icon at the top right hand corner
   3. Click "User Settings"
   4. Click "SMTP Credentials" at the bottom left hand corner
   5. Click "Generate SMTP Credentials"
   6. Copy Username and Password for your records. They will not be shown again once the popup menu disappears
2. Create an Approved Sender
   1. Click OCI messaging/email/senders
   2. Set OCI region to "US East (Ashburn)"
   3. Pick a desired "Compartment" from left hand column
   4. Click "Create Approved Sender"
   5. Enter sauron-alert@sauron.us-ashburn-1.oracledx.com
   6. Click "Create Approved Sender" from the popup menu. Note:
      • OCI is strict about Email Domain of Approved Sender
      • The following Email Domains are not allowed:
        • @oracle.com, @*.oraclevcn.com, or @oraclecloud.com
        • @gmail.com, @hotmail.com, @yahoo.com, @icloud.com or any public mail providers
      • @sauron.us-ashburn-1.oracledx.com should work
      • You may try Email Domain that is under your control, but YMMV
      • If you're in doubt, please contact #sauron-support for help

3. Update Alertmanager configuration

   1. Login to https://api.handu-phx.handu.developers.oracledx.com/#/Alertmanager/getAlertmanagerConfig
   2. Click "Try it out"
   3. Click "Execute"
   4. Copy existing Alertmanager configuration
   5. Go to https://api.handu-phx.handu.developers.oracledx.com/#/Alertmanager/putAlertmanagerConfig
   6. Click "Try it out"
   7. Paste existing Alertmanager configuration

   8. Massage the email_configs section (listed below) into existing Alertmanager configuration

      route:
        receiver: "my_alert_receivers"
        group_by: ['alertname']
        group_wait: 30s
        group_interval: 1m
        repeat_interval: 3m
      receivers:
      - name: "my_alert_receivers"
        email_configs:
        - from: '<Approved Sender, e.g. sauron-alert>@sauron.us-ashburn-1.oracledx.com'
          to: 'first.last@oracle.com'
          smarthost: smtp.us-ashburn-1.oraclecloud.com:587
          auth_username: '<SMTP Username, e.g. ocid1.user.oc1.xxxxxxxxxx.com>'
          auth_password: '<SMTP Password, e.g. somepassword>'
          #headers:
            #subject: "You have {{ .Alerts.Firing | len }} firing alerts"
          #html: '{{ template "email.key" . }}'
          send_resolved: true
      #templates:
      #- '/etc/alertmanager/templates/email.tmpl'
      

   9. Click "Execute"

4. Send a test alert to Alertmanager

   curl -v -u sauron:mypassword \
   -d '[{"labels": {"alertname": "test-alert-1", "severity": "warning"}}]' \
   https://alertmanager.handu-phx.handu.developers.oracledx.com/api/v1/alerts
   

An email should be delivered to "first.last@oracle.com":

1. (Optional) Customize the subject and body of alert emails
   1. Go to https://api.handu-phx.handu.developers.oracledx.com/#/Alertmanager/putAlertmanagerTemplate
   2. Click "Try it out"
   3. Enter email.tmpl as Alertmanager template filename
   4. Modify "Email Template" (listed below) to your liking
   5. Copy and paste modified email template, including {{ define "email.key" }} and {{ end }}
   6. Click "Execute"
   7. Enable email template by repeating Step: Update Alertmanager configuration above with # (comment) signs removed from the email_configs section
   8. Resend test alert by repeating Step: Send a test alert to Alertmanager above

{{ define "email.key" }}
  <!--
  Style and HTML derived from https://github.com/mailgun/transactional-email-templates
  The MIT License (MIT)
  Copyright (c) 2014 Mailgun
  Permission is hereby granted, free of charge, to any person obtaining a copy
  of this software and associated documentation files (the "Software"), to deal
  in the Software without restriction, including without limitation the rights
  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
  copies of the Software, and to permit persons to whom the Software is
  furnished to do so, subject to the following conditions:
  The above copyright notice and this permission notice shall be included in all
  copies or substantial portions of the Software.
  THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
  SOFTWARE.
  -->
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml">
  <head>
  <meta name="viewport" content="width=device-width" />
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <title>{{ template "__subject" . }}</title>
  <style>
  /* -------------------------------------
      GLOBAL
      A very basic CSS reset
  ------------------------------------- */
  * {
    margin: 0;
    font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
    box-sizing: border-box;
    font-size: 14px;
  }

  img {
    max-width: 100%;
  }

  body {
    -webkit-font-smoothing: antialiased;
    -webkit-text-size-adjust: none;
    width: 100% !important;
    height: 100%;
    line-height: 1.6em;
    /* 1.6em * 14px = 22.4px, use px to get airier line-height also in Thunderbird, and Yahoo!, Outlook.com, AOL webmail clients */
    /*line-height: 22px;*/
  }

  /* Let's make sure all tables have defaults */
  table td {
    vertical-align: top;
  }

  /* -------------------------------------
      BODY & CONTAINER
  ------------------------------------- */
  body {
    background-color: #f6f6f6;
  }

  .body-wrap {
    background-color: #f6f6f6;
    width: 100%;
  }

  .container {
    display: block !important;
    max-width: 600px !important;
    margin: 0 auto !important;
    /* makes it centered */
    clear: both !important;
  }

  .content {
    max-width: 600px;
    margin: 0 auto;
    display: block;
    padding: 20px;
  }

  /* -------------------------------------
      HEADER, FOOTER, MAIN
  ------------------------------------- */
  .main {
    background-color: #fff;
    border: 1px solid #e9e9e9;
    border-radius: 3px;
  }

  .content-wrap {
    padding: 30px;
  }

  .content-block {
    padding: 0 0 20px;
  }

  .header {
    width: 100%;
    margin-bottom: 20px;
  }

  .footer {
    width: 100%;
    clear: both;
    color: #999;
    padding: 20px;
  }
  .footer p, .footer a, .footer td {
    color: #999;
    font-size: 12px;
  }

  /* -------------------------------------
      TYPOGRAPHY
  ------------------------------------- */
  h1, h2, h3 {
    font-family: "Helvetica Neue", Helvetica, Arial, "Lucida Grande", sans-serif;
    color: #000;
    margin: 40px 0 0;
    line-height: 1.2em;
    font-weight: 400;
  }

  h1 {
    font-size: 32px;
    font-weight: 500;
    /* 1.2em * 32px = 38.4px, use px to get airier line-height also in Thunderbird, and Yahoo!, Outlook.com, AOL webmail clients */
    /*line-height: 38px;*/
  }

  h2 {
    font-size: 24px;
    /* 1.2em * 24px = 28.8px, use px to get airier line-height also in Thunderbird, and Yahoo!, Outlook.com, AOL webmail clients */
    /*line-height: 29px;*/
  }

  h3 {
    font-size: 18px;
    /* 1.2em * 18px = 21.6px, use px to get airier line-height also in Thunderbird, and Yahoo!, Outlook.com, AOL webmail clients */
    /*line-height: 22px;*/
  }

  h4 {
    font-size: 14px;
    font-weight: 600;
  }

  p, ul, ol {
    margin-bottom: 10px;
    font-weight: normal;
  }
  p li, ul li, ol li {
    margin-left: 5px;
    list-style-position: inside;
  }

  /* -------------------------------------
      LINKS & BUTTONS
  ------------------------------------- */
  a {
    color: #348eda;
    text-decoration: underline;
  }

  .btn-primary {
    text-decoration: none;
    color: #FFF;
    background-color: #348eda;
    border: solid #348eda;
    border-width: 10px 20px;
    line-height: 2em;
    /* 2em * 14px = 28px, use px to get airier line-height also in Thunderbird, and Yahoo!, Outlook.com, AOL webmail clients */
    /*line-height: 28px;*/
    font-weight: bold;
    text-align: center;
    cursor: pointer;
    display: inline-block;
    border-radius: 5px;
    text-transform: capitalize;
  }

  /* -------------------------------------
      OTHER STYLES THAT MIGHT BE USEFUL
  ------------------------------------- */
  .last {
    margin-bottom: 0;
  }

  .first {
    margin-top: 0;
  }

  .aligncenter {
    text-align: center;
  }

  .alignright {
    text-align: right;
  }

  .alignleft {
    text-align: left;
  }

  .clear {
    clear: both;
  }

  /* -------------------------------------
      ALERTS
      Change the class depending on warning email, good email or bad email
  ------------------------------------- */
  .alert {
    font-size: 16px;
    color: #fff;
    font-weight: 500;
    padding: 20px;
    text-align: center;
    border-radius: 3px 3px 0 0;
  }
  .alert a {
    color: #fff;
    text-decoration: none;
    font-weight: 500;
    font-size: 16px;
  }
  .alert.alert-warning {
    background-color: #E6522C;
  }
  .alert.alert-bad {
    background-color: #D0021B;
  }
  .alert.alert-good {
    background-color: #68B90F;
  }

  /* -------------------------------------
      INVOICE
      Styles for the billing table
  ------------------------------------- */
  .invoice {
    margin: 40px auto;
    text-align: left;
    width: 80%;
  }
  .invoice td {
    padding: 5px 0;
  }
  .invoice .invoice-items {
    width: 100%;
  }
  .invoice .invoice-items td {
    border-top: #eee 1px solid;
  }
  .invoice .invoice-items .total td {
    border-top: 2px solid #333;
    border-bottom: 2px solid #333;
    font-weight: 700;
  }

  /* -------------------------------------
      RESPONSIVE AND MOBILE FRIENDLY STYLES
  ------------------------------------- */
  @media only screen and (max-width: 640px) {
    body {
      padding: 0 !important;
    }

    h1, h2, h3, h4 {
      font-weight: 800 !important;
      margin: 20px 0 5px !important;
    }

    h1 {
      font-size: 22px !important;
    }

    h2 {
      font-size: 18px !important;
    }

    h3 {
      font-size: 16px !important;
    }

    .container {
      padding: 0 !important;
      width: 100% !important;
    }

    .content {
      padding: 0 !important;
    }

    .content-wrap {
      padding: 10px !important;
    }

    .invoice {
      width: 100% !important;
    }
  }
  </style>
  </head>

  <body itemscope itemtype="http://schema.org/EmailMessage">

  <table class="body-wrap">
    <tr>
      <td></td>
      <td class="container" width="600">
        <div class="content">
          <table class="main" width="100%" cellpadding="0" cellspacing="0">
            <tr>
              {{ if gt (len .Alerts.Firing) 0 }}
              <td class="alert alert-warning">
              {{ else }}
              <td class="alert alert-good">
              {{ end }}
                {{ .Alerts | len }} alert{{ if gt (len .Alerts) 1 }}s{{ end }} for {{ range .GroupLabels.SortedPairs }}
                  {{ .Name }}={{ .Value }}
                {{ end }}
              </td>
            </tr>
            <tr>
              <td class="content-wrap">
                <table width="100%" cellpadding="0" cellspacing="0">
                  <tr>
                    <td class="content-block">
                      <a href='{{ template "__alertmanagerURL" . }}' class="btn-primary">View in {{ template "__alertmanager" . }}</a>
                    </td>
                  </tr>
                  {{ if gt (len .Alerts.Firing) 0 }}
                  <tr>
                    <td class="content-block">
                      <strong>[{{ .Alerts.Firing | len }}] Firing</strong>
                    </td>
                  </tr>
                  {{ end }}
                  {{ range .Alerts.Firing }}
                  <tr>
                    <td class="content-block">
                      <strong>Labels</strong><br />
                      {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
                      {{ if gt (len .Annotations) 0 }}<strong>Annotations</strong><br />{{ end }}
                      {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
                      <a href="{{ .GeneratorURL }}">Source</a><br />
                    </td>
                  </tr>
                  {{ end }}

                  {{ if gt (len .Alerts.Resolved) 0 }}
                    {{ if gt (len .Alerts.Firing) 0 }}
                  <tr>
                    <td class="content-block">
                      <br />
                      <hr />
                      <br />
                    </td>
                  </tr>
                    {{ end }}
                  <tr>
                    <td class="content-block">
                      <strong>[{{ .Alerts.Resolved | len }}] Resolved</strong>
                    </td>
                  </tr>
                  {{ end }}
                  {{ range .Alerts.Resolved }}
                  <tr>
                    <td class="content-block">
                      <strong>Labels</strong><br />
                      {{ range .Labels.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
                      {{ if gt (len .Annotations) 0 }}<strong>Annotations</strong><br />{{ end }}
                      {{ range .Annotations.SortedPairs }}{{ .Name }} = {{ .Value }}<br />{{ end }}
                      <a href="{{ .GeneratorURL }}">Source</a><br />
                    </td>
                  </tr>
                  {{ end }}
                </table>
              </td>
            </tr>
          </table>

          <div class="footer">
            <table width="100%">
              <tr>
                <td class="aligncenter content-block"><a href='{{ .ExternalURL }}'>Sent from {{ template "__alertmanager" . }}</a></td>
              </tr>
            </table>
          </div></div>
      </td>
      <td></td>
    </tr>
  </table>

  </body>
  </html>
{{ end }}

• Notice the subject and body of email had changed.

Defining Email Integration for Grafana Alerts

Pre-requisite 1: the following steps require an OCI tenancy. If you or your development team don’t have access to an OCI tenancy, then you can’t setup approved email senders in OCI, and as a result, you can’t send email alerts from Grafana.

Pre-requisite 2: please contact #sauron-support to enable SMTP for your Grafana endpoint. Otherwise the following steps won't take effect.

1. Generate SMTP Credential
   1. Login OCI console
   2. Click "Profile" icon at the top right hand corner
   3. Click "User Settings"
   4. Click "SMTP Credentials" at the bottom left hand corner
   5. Click "Generate SMTP Credentials"
   6. Copy Username and Password for your records that will be used in subsequent steps. They will not be shown again once the popup menu disappears
2. Create an Approved Sender
   1. Click OCI messaging/email/senders
   2. Set OCI region to "US East (Ashburn)"
   3. Pick a desired "Compartment" from left hand column
   4. Click "Create Approved Sender"
   5. Enter sauron-alert@sauron.us-ashburn-1.oracledx.com
   6. Click "Create Approved Sender" from the popup menu. Note:
      • OCI is strict about Email Domain of Approved Sender
      • The following Email Domains are not allowed:
        • @oracle.com, @*.oraclevcn.com, or @oraclecloud.com
        • @gmail.com, @hotmail.com, @yahoo.com, @icloud.com or any public mail providers
      • @sauron.us-ashburn-1.oracledx.com should work
      • You may try Email Domain that is under your control, but YMMV
      • If you're in doubt, please contact #sauron-support for help

3. Update Grafana Email alerts configuration

   1. Login to https://api.handu-phx.handu.developers.oracledx.com/#/Grafana/putGrafanaEmailAlertsConfig
   2. Click "Try it out"
   3. Fill in:
      • smtpFrom: sauron-alert@sauron.us-ashburn-1.oracledx.com
      • smtpHost: smtp.us-ashburn-1.oraclecloud.com:587
      • smtpUser: ocid1.user.oc1..aaaaaaaa...@ocid1.tenancy.oc1..aaaaaaaa...
      • smtpPassword: ******

   To get smtpUser go to Identity->Users->User Details->SMTP Credentials, copy the username of the SMTP Credentials that were created in step 1. smtpPassword you would have noted during step 1.

   1. Click "Execute"
   2. Grafana will be restarted to apply your SMTP settings
   3. Create Grafana Email notification channel
   4. Login to https://grafana.handu-phx.handu.developers.oracledx.com/alerting/notification/new
   5. Follow the screenshot below to create a new email notification channel
   6. Click "Test", a test notification email should be delivered to "first.last@oracle.com" shortly
   7. (Optional) Customize Grafana Email Alerts Template
   8. Goto https://api.handu-phx.handu.developers.oracledx.com/#/Grafana/getGrafanaEmailAlertsTemplate and get a copy of default email template
   9. Customize the template to your liking
   10. Goto https://api.handu-phx.handu.developers.oracledx.com/#/Grafana/putGrafanaEmailAlertsTemplate
   11. Click "Try it out"
   12. Put your custom email template into the "body" field
   13. Click "Execute"
   14. Grafana will be restarted to apply your custom email template
   15. If necessary, use https://api.handu-phx.handu.developers.oracledx.com/#/Grafana/deleteGrafanaEmailAlertsTemplate to reset email template back to default

Defining Slack Integration for Alertmanager Alerts

For Slack integration, you need to provide a Slack Webhook API URL in the Alertmanager configuration. If proddev-paas-fmw is your Oracle Workspace then please contact sauron-support and we can provide you the Webhook API URL.

If you create a private workspace on Slack, please use a personal email address, e.g. @gmail.com, because Oracle is a paid customer, and an @oracle.com email address wont work.

To create a private Slack workspace:

1. Click on workspaces, click "Add Workspaces" . Use new email address say your @gmail.com address.
2. It will send some code to your personal email, and prompt for workspace URL.
3. Select something that is available for e.g. <firstNameLastName>-test.slack.com. And you should be set
4. You would need to invite your colleagues to your private workspace with their email addresses.
5. Go to Administration | Manage Apps.
6. Click on Custom Integration, Incoming Webhooks, Add Configuration
7. On this screen you should see the Webhook URL, Click on Copy URL. Use this in api_url field below in the Alertmanager configuration.

For official Oracle Slack workspaces:

1. You do the same above steps but you would need approval from the Slack Administrator for that workspace before you get the Webhook URL.
2. On the Manage Apps. Select For e.g. https://proddev-paas-fmw.slack.com/apps/A0F7XDUAZ-incoming-webhooks where proddev-paas-fmw is your Oracle Workspace
3. Click on Request to Install
4. Add a valid message say Posts to #myteam-alerts public channel for myteam alerts from Alertmanager and hit Submit Request
5. Once your request is approved , you should get the Webhook URL

global:
  slack_api_url: https://hooks.slack.com/services/xxx/xxxx/xxxx
route:
  receiver: Service-Slack
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 3m
receivers:
- name: Service-Slack
  slack_configs:
  - icon_url: https://alertmanager.handu-phx.handu.developers.oracledx.com
    send_resolved: true
    link_names: true
    title_link: https://alertmanager.handu-phx.handu.developers.oracledx.com/#/alerts?receiver={{ .Receiver }}
    channel: '#your-channel'
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Sauron Monitoring Event Notification'
    text: >-
        {{ range .Alerts }}
           *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          *Description:* {{ .Annotations.description }}
          *AlertURL:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
          *Details:*
          {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
          {{ end }}
        {{ end }}

To send a test alert to your Alertmanager:

1. Update your Alertmanager configuration:

   curl -u sauron:mypassword \
   -i -T alertmanager.yml \
   https://api.handu-phx.handu.developers.oracledx.com/#/Alertmanager/putAlertmanagerConfig
   

2. Send test alert:

   curl -u sauron:mypassword \
   -d '[{"annotations": {"summary": "Firing a test alert with customized summary"}, "description": "Some test slack description", "generatorURL": "https://alertmanager.handu-phx.handu.developers.oracledx.com","labels": {"alertname": "SlackClientDemo", "severity": "slack", "job": "TestSlackJob", "instance": "192.0.2.2:9173", "health_state": "healthy", "env": "https://alertmanager.handu-phx.handu.developers.oracledx.com"}}]' \
   https://alertmanager.handu-phx.handu.developers.oracledx.com/api/v1/alerts
   

3. A message should arrive on the Slack channel defined in slack_configs.

Defining Multiple Routing to Ocean/Slack/Email for Metric Alerts

You could have a multi routing schema based on the severity of your Alerts

• All Alerts with severity Critical or by default go to Ocean
• All Alerts with some label which matches a regular expression go to different Ocean
• All Alerts with severity Warning go to Slack
• All Alerts with severity Info go to Email

global:
  resolve_timeout: 3m
  smtp_smarthost: smtp.us-ashburn-1.oraclecloud.com:587
  smtp_from: sauron-alert@sauron.us-ashburn-1.oracledx.com
  smtp_auth_username: ocid1.user.oc1.xxxxxxxxxxx.com
  smtp_auth_password: somepasswd
  slack_api_url: https://hooks.slack.com/services/xxxxx/xxxxx/xxxxxxxx
route:
  receiver: Service-Production
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 1m
  repeat_interval: 3m
  routes:
  - receiver: CustomMatchExpression
    group_by: ['env']
    group_wait: 30s
    group_interval: 1m
    repeat_interval: 3m
    match_re:
      env: (someString).*
  - receiver: email-prod
    group_by: ['alertname']
    match:
      severity: info
  - receiver: slack-prod
    group_by: ['alertname']
    match:
      severity: warning
receivers:
- name: Service-Production
  webhook_configs:
  - xxxxxxxxxxxxxxxxxx
- name: CustomMatchExpression
  webhook_configs:
  - xxxxxxxxxxxxxxxxxx
- name: email-prod
  email_configs:
  - to: first.last@oracle.com
    send_resolved: true
- name: slack-prod
  slack_configs:
  - icon_url: https://alertmanager.handu-phx.handu.developers.oracledx.com
    send_resolved: true
    link_names: true
    title_link: https://alertmanager.handu-phx.handu.developers.oracledx.com/#/alerts?receiver={{ .Receiver }}
    channel: '#your-channel'
    title: '[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] Sauron Monitoring Event Notification'
    text: >-
        {{ range .Alerts }}
           *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
          *Description:* {{ .Annotations.description }}
          *AlertURL:* <{{ .GeneratorURL }}|:chart_with_upwards_trend:> *Runbook:* <{{ .Annotations.runbook }}|:spiral_note_pad:>
          *Details:*
          {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
          {{ end }}
        {{ end }}

Alerting for Logs/Events - OpenSearch Alerting

Overview

OpenSearch Alerting is an open source observability tool for alerting based on data stored in OpenSearch.

To get started, you should read the docs for OpenSearch Alerting and also read the docs for OpenSearch Notifications Channels.

Here's a quick end-to-end walkthrough of how to create an OpenSearch alert:

• Create and update a monitor, which is a job that runs inside OpenSearch cluster at a pre-defined schedule and queries OpenSearch APIs or indices.

• Add one or more triggers, which define monitor job conditions that, once met, trigger alerts.

• Add one or more actions, which are what will happen once alert triggered, such as, sending notification to predefined notification channel.

NOTE: if your notification channel is going to be Alertmanager, then the "Message" field of an action need to follow a specific format, see comments below for details.

[{ // the round and curly brackets are required here
  "labels": { // the labels block is required here
    "alertname": "{{ctx.monitor.name}}", // the label - alertname is required
    "severity": "{{ctx.trigger.severity}}" // the label - severity is required
  },
  "annotations": { // the annotations block is optional, but highly recommended
    "summary": "Monitor {{ctx.monitor.name}} - Trigger: {{ctx.trigger.name}} fired. Please investigate the issue!",
    "playbook":  "xxxxxxxx"
  },
  "generatorURL": "xxxxxxxx" // the generatorURL is optional, but highly recommended
}] // the round and curly brackets are required here

• Create and edit a notification channel, which will be used to deliver notifications to incident management service, like Ocean.

NOTE1: the Webhook URL of your Alertmanager is "https://alertmanager.handu-phx.handu.developers.oracledx.com/api/v2/alerts"

NOTE2: make sure to add all three Webhook headers. Specifically, the value of "Authorization" header should be in the format of "Basic base64Value", where "base64Value" is output of echo -n your_sauron_reporter:password | base64, echo -n is required here.