Configuring Alerts

An alert may happen when the state of a check has been calculated from a result.

Algorithms

There are different algorithms to evaluate the alert conditions.

State change

If the current state differs from the previous one.

State recurrence

If the current state is not OK and lasts for more than X occurences.

State flap

If the probe has state changing more than X times in the last Y seconds.

Channels

When an alert occurs, it sends a notification through a "Channel". Channels can be created and configured in the Create Alert dialog. The configuration of the Channel depends on the type:

Email

Alerts can be notified via email. The outgoing email server (SMTP) settings are configured on the server.

Configuration options

  • dest: the To: field of the sent email, a comma-separated list of email addresses
  • cc: the Cc: field of the sent email, a comma-separated list of email addresses (optional)

Slack

Alerts can be notified on Slack, using an incoming webhook. Create an Slack incoming webhook by following the official instructions.

Configuration options

  • webhook_url: the URL of the incoming webhook for your Slack integration

PagerDuty

Alerts can be forwarded to PagerDuty to integrate with your operations management platform.

Configuration options

  • from: the email address of the person raising this incident
  • token: a PagerDuty API token
  • service: the ID of the PagerDuty service this incident is raised for

SMS

SMS alerting is provided via Twilio. You need a Twilio account to create an SMS alert Channel.

Configuration options

  • account-sid: a Twilio account SID
  • token: a Twilio Auth token
  • from: a Twilio phone number or short code that sends this message
  • messaging-service-sid: a Twilio Messaging service SID
  • to: a Phone number to send the message to

Note: Only one of from or messaging-service-sid is required.

HTTP

Alerts can be sent to an arbitrary HTTP endpoint. This Channel is used to build custom integrations with other 3rd-party services or with your own in-house tools.

Configuration options

  • url: the URL of the HTTP endpoint to send the alert to

The HTTP Channel POSTs a JSON representation of the Alert to the URL. The following properties can be found on the JSON:

  • title: a title for the Alert
  • message: why the Alert was triggered
  • target: the display name of the Target that exhibits abnormal conditions
  • agent: the display name of the Agent that ran the Probe on the Target
  • probe: the name of the Probe that got abnormal results
  • state: the current state, WARNING or CRITICAL
  • reason: the user-supplied reason in the check(s) that weren't validated
  • critical: true if state is CRITICAL
  • warning: true if state is WARNING