Create a Threshold Metric Alert

Metric monitors analyze continuous streams of data. When collecting metrics through Kloudfuse or APIs, you can configure alerts to trigger on specified conditions.

To create a threshold alert for a metric, complete these tasks:

Start defining a metric alert

  1. In the Alerts interface, click Creae New Alert (Create New Alert).

  2. In the initial interface for creating alerts, Kloudfuse guides you to select the appropriate alert type.

    Choose Metric.

  3. The Create Metric Alert interface appears.

Choose the detection method

Metric alerts have these detection options:

Threshold Alert

Select Threshold alert

This is the alert type described in this document.

Change Alert

select Change alert

Pick metric

  1. Next to the first query, marked by a, click the text field to select a metric from the drop-down.

    Select metric

  2. The chart of the metric over the time period appears.

You can proceed with these optional configurations:

Add Filters

  1. To add a new filter, select a label and add it to the metric view.

    Next to the From(From) tag, click the Add (Add) icon.

  2. In the expanded view, click the first Select (Select) option, and choose the new label from the drop-down.

  3. Choose the comparison operator:

    =~

    Regex expression equal

    !~

    Regex expression not equal

    !=

    Not equal

    =

    Equal (default)

  4. Finally, click the second Select (Select) option, and choose the label value from the drop-down.

  5. To add another filter, click the Add (Add) icon, and specify the new label.

    Remember that the filter selections combine using AND logic.

  6. To remove a filter, click the Delete (Delete) icon that corresponds to the label/value pair that you plan to remove.

  7. The filter no longer appears on the page.

Add Aggregations

  1. In the aggregation section, click avg, and select another aggregation calculation from the drop-down.

  2. The chart changes to show the newly selected aggregation of results.

  3. To aggregate over another dimension, in the metrics selector specify the aggregation by clicking Select, and choosing another metric option from the drop-down.

  4. To remove the aggregation, in the aggregation section click the Remove (Remove) icon.

Add Functions

Add Function

  1. To add a new function to the graph, click the Sigma (Sigma) icon at the end of the query specification.

  2. In the drop-down, either search for a function name, or select an existing function category, and then pick the function name.

  3. Configure the parameters of the function, if any.

  4. To remove a function, click Remove function (Remove function) icon, next to the function you want to drop.

Add More Queries

  1. To add a query, click Add Query button (Add Query).

  2. Notice that a new query B (B) appears under query A (A), and it is a duplicate.

    This is intentional, and makes it is easier to perform calculations across the time series retrieved by the two queries.

  3. Change the selected filter in query B (B).

  4. To remove one of the queries, click the Remove (Remove) icon that corresponds the to query you plan to remove.

Add Formulas

Add formula

You can combine metrics over facets or labels into formulas, to get additional insights.

  1. To add a formula, click Add Formula button (Add Formula).

  2. The empty formula option appears in interface as line 1 (1), under the query lines.

  3. Type the formula in the text box next to label 1 (1).

  4. To remove a formula, click Delete (Delete) icon that corresponds to the formula you plan to remove.

  5. The formula is no longer in the interface.

Change step size

  1. Click the Steps option of the metric.

  2. Notice the default step size, and change it to what you require.

  3. Observe how the graph for the metric changes with new step size.

  4. If you click the Delete (Delete) icon, the step size reverts to the default.

Set condition

There are some differences in the process of setting the alerting condition, depending on the detection method.

  1. Define the condition.

    Condition for Threshold Metric Alerts

    1. From the first drop-down, select the condition function: Mean, Min, Max, Sum, Count, or Last. The default is Min.

    2. From the second drop-down, select the evaluated query (Query (a), Query (b), and so on) or formula (Formula (1), Formula (2), and so on).

    3. From the third drop-down, select the comparison operator: above (default), below, equal to, not equal to, above or equal to, or below or equal to.

    4. [Optional] In the text box, specify the unit of the measurement.

    5. In the fourth (last) drop-down, specify the time interval: 5 minutes, 10 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, 4 hours, 1 day, 2 days, 1 week, or 1 month.

      This interval sets how far back each evaluation reads data. It does not delay the alert. To fire only after the condition has held for the whole interval, see Fire only after a condition persists. . Expand the Configure no data and error handling option.

      + image::alerts-no-data-error-handling.svg[]

      +

      No data

      What to do when data is missing.

      In the first drop-down, select one of the options: Alerting, No Data, OK (default), or Keep Last State.

      Metric threshold alerts only, plus metric outlier alerts in the new Kloudfuse UI: Evaluate as Zero treats a series' missing samples as 0, so the series is still compared against the threshold. Use it only for a series that exists only while there is a problem. For a series that reports continuously, a collection gap would count as 0. Other alert types, including log alerts and metric change, anomaly, and forecast alerts, don’t have this option. To use it so that an alert fires only after a condition persists, see Fire only after a condition persists.

      Error handling

      What to do when there is an execution error, or a timeout.

      In the second drop-down, select one of the options: Alerting, OK, Error, or Keep Last State (default).

Fire only after a condition persists

This section applies only to the Threshold detection method. Change, Outliers, Anomaly, and Forecast alerts don’t support it.

Sometimes an alert should fire only when a condition has held for a period of time. For example, you want to know when a pod has been in ImagePullBackOff for 15 minutes, not when it gets stuck for a moment.

The alert window does not do this on its own. The window is Over the last in the new Kloudfuse UI and threshold over in the existing UI. It sets how far back each evaluation reads data. It is not a pending period, and it does not delay the alert.

Instead, use the Min reducer with is above. Min turns the window into its lowest value, so the alert fires only when every value in the window is above the threshold. With Last, the alert fires as soon as the latest value crosses the threshold, even on a brief spike. The new Kloudfuse UI selects Last by default. The existing UI selects Min by default. There is no equivalent hold for is below conditions today.

What else you set depends on how the series reports:

The series reports continuously

The series exists whether or not there is a problem, for example CPU usage, memory, or request latency. Use Min alone. Don’t turn on Evaluate as Zero: a collection gap would count as 0 and suppress a valid alert.

The series exists only while there is a problem

Examples are a count of waiting containers that exists only while a pod is stuck, or a series for a node that just joined. Min reads only the samples that exist, so when such a series appears, Min sees only a few samples, all above the threshold, and the alert fires at its first evaluation.

Use Min plus Evaluate as Zero. Evaluate as Zero fills the missing samples of each series within the window with 0. Because 0 doesn’t meet the condition, the alert then waits until the condition has held for the whole window. This works only with is above and a threshold of 0 or more. With a negative threshold, or with is above or equal 0, the filled-in zeros meet the condition.

Configure the alert

  1. In Set alert condition, set the condition row to When Min of query <your query> is above Threshold <value>.

  2. Under Optional settings, expand Evaluation frequency.

    1. In Over the last, select how long the condition must hold, for example 15m.

    2. In Evaluate every, select how often to check, for example 1m.

  3. Only if the series exists only while there is a problem: expand No data and error handling, and in If there is no data, select Evaluate as Zero. With this setting, use is above, not is above or equal, and a threshold of 0 or more.

Evaluate as Zero stays in place when you edit and save the alert in either UI.

Example: pod stuck in ImagePullBackOff for 15 minutes

This example uses a series that exists only while there is a problem. To alert when a pod has been in ImagePullBackOff for 15 minutes, use these settings:

  • Query: kubernetes_state_container_status_report_count_waiting, filtered to reason="ImagePullBackOff", aggregated by kube_cluster_name, kube_namespace, and pod_name.

  • Condition: Min is above 0.

  • Window: 15 minutes.

  • Evaluate every: 1 minute.

  • If there is no data: Evaluate as Zero.

The waiting count exists only while a container is waiting. Without Evaluate as Zero, the alert fires about one minute after a pod gets stuck. With it, the alert fires about 15 to 16 minutes after a pod gets stuck, and separately for each stuck pod, identified by its cluster, namespace, and pod name. The alert resolves at the first evaluation after the pod recovers.

Things to know

  • Above conditions only. The hold works only with Min and is above. There is no equivalent hold for is below conditions today. Don’t use Evaluate as Zero with them: a missing sample counts as 0, which makes gaps look like breaches. The hold also doesn’t work with Last, Mean, Sum, or the other reducers.

  • Timing. The alert fires about one window plus up to one evaluation interval after the condition starts. For example, with a 15-minute window and a 5-minute evaluation interval, it fires about 15 to 20 minutes after onset. For tighter timing, use a shorter evaluation interval.

  • No hold on recovery. With no Alert recovery threshold set, the alert resolves at the first evaluation where any value in the window is back below the threshold, or, with Evaluate as Zero, missing. If you set an Alert recovery threshold, the alert resolves only when the value passes back through the recovery threshold.

  • Alert types. This applies to metric threshold alerts. Evaluate as Zero is available for metric threshold alerts in both UIs, and for metric outlier alerts in the new Kloudfuse UI. Log, APM, RUM, and event alerts don’t have it.

With Evaluate as Zero, also note:

  • Zero must not meet the condition. Use is above with a threshold of 0 or more. With a negative threshold, or with is above or equal 0, the filled-in zeros meet the condition, so gaps look like breaches and the alert fires early.

  • Gaps count as 0. Any missing sample of a series within the window counts as 0 and keeps an is above alert from firing. A series that reports irregularly, or that flaps, never fires. Don’t use it for a series that reports continuously.

Add details

  1. In the Folder name drop-down, select one of the folders.

    alerts choose folder

    Alternatively, click the Create new folder (Create new folder) icon.

    Then, in the Create Folder dialog, specify Folder name, and click Create new folder (Create new folder).

    create folder dialog

  2. In Rule name, specify the name of your rule.

  3. [Optional] In Title, enter the title of your rule.

  4. [Optional] In Runbook URL, enter the location of the runbook that specifies how to handle the alert.

  5. [Optional] Under Description, specify the purpose of the rule, what conditions it should catch, and so on.

  6. [Optional] Under Custom labels, click Add (plus), and specify the custom label name, and its value.

    To add more custom labels, click Add (plus), and specify more custom labels.

    To remove a custom label, click Trash (Trash) next to it.

    alerts custom labels

  7. [Optional] Under Custom annotations, click Add (plus), and specify the custom annotation name, and its value.

    To add more custom annotations, click Add (plus), and specify more custom annotations.

    To remove a custom annotation, click Trash (Trash) next to it.

    alerts custom annotations

    Use annotations to show different alerts and their visualizations.

Add contacts

Choose contact points

  1. Choose one or more of the existing contacts from the drop-down, so they can receive the alert.

    alerts choose contact points

  2. To remove a contact you previously selected from the alert notification, click delete (Delete) icon on their tile.

    alerts remove contact point

  3. To remove all contacts that you previously selected, click the delete (Delete) icon on the right side of the drop-down.

    alerts remove contact point all

  4. Below the selector, the Routing preview strip reports where the alert actually delivers. See Notification Routing Preview.

Create new contact points

  1. Click Create New Contact Points (Create New Contact Points).

    alerts create contact points

  2. Kloudfuse opens the Create Contact points interface.

    Specify, configure, test, and save new contact points. See Contact Points for details.

Create rule

  1. To finalize, click Create Rule (Create Rule).

  2. Depending on your choice of calculation when you Set condition (query a, query b, formula 1, formula 2, and so on) Kloudfuse generates a Confirm Query Selection dialog.

  3. In the Confirm Query Selection dialog, click Confrim and Create(Confirm and Create)

  4. You get a confirmation message.