Derived Metrics

Table of Contents

When to use derived metrics
Derived metrics vs. metric shaping rules
Access derived metrics
Create a derived metric
Query the result
Edit, pause, or delete a rule
Define rules in Helm values
Troubleshooting
See also

Derived metrics let you precompute a PromQL expression on a fixed interval and store the result as a brand-new metric you can query directly. Behind the scenes, Kloudfuse evaluates your expression every interval and writes a new time series so that downstream dashboards and alerts no longer have to re-execute the expensive original query each time they refresh.

If you have a heavy sum by (…) or histogram_quantile(…) query that you keep referencing in dashboards or alerts, turning it into a derived metric makes that query cheap to read.

When to use derived metrics

Use derived metrics when one or more of the following is true:

A PromQL expression appears in multiple dashboards or alerts, and re-evaluating it on every refresh is slow or expensive.
An aggregation collapses thousands of high-cardinality series into a small fixed set — for example, "requests per cluster" computed from per-pod counters.
A histogram percentile (histogram_quantile) is queried frequently, and you want a single precomputed series instead of scanning all bucket series at read time.

Don’t use derived metrics for one-off ad-hoc queries: the cost of running the rule every interval may exceed the savings.

Derived metrics vs. metric shaping rules

Both features produce a new precomputed metric, but they solve different problems.

Aspect	Derived metrics	Metric shaping rules
Goal	Cache an arbitrary PromQL expression so dashboards and alerts read it cheaply.	Reduce a metric’s cardinality by aggregating it before storage, replacing the original series.
Source	Any PromQL — including rate, histogram_quantile, multi-query formulas.	A single source metric, aggregated by a fixed set of labels.
Original series preserved?	Always. Derived metrics are additive — your source metric is untouched.	No. A shaping rule created in the UI replaces the source metric with its output.
When the result appears	At the next rule evaluation (after the configured interval).	At the next rule evaluation (after the configured interval).
Where to find it	menu:Metrics[Derived metrics]	menu:Admin[Settings > Shaping rules]

Aspect

Derived metrics

Metric shaping rules

Goal

Cache an arbitrary PromQL expression so dashboards and alerts read it cheaply.

Reduce a metric’s cardinality by aggregating it before storage, replacing the original series.

Source

Any PromQL — including rate, histogram_quantile, multi-query formulas.

A single source metric, aggregated by a fixed set of labels.

Original series preserved?

Always. Derived metrics are additive — your source metric is untouched.

No. A shaping rule created in the UI replaces the source metric with its output.

When the result appears

At the next rule evaluation (after the configured interval).

Where to find it

menu:Metrics[Derived metrics]

menu:Admin[Settings > Shaping rules]

If you want to reduce the cardinality of an existing high-volume metric, use Metric Shaping Rules. If you want to precompute and cache an expression result without touching the original metric, you’re in the right place — keep reading.

Access derived metrics

From the left navigation, click Metrics.
Select Derived metrics from the dropdown menu.

The Derived metrics page opens. It is empty until you create the first rule, and has:

Create derived metric: Top right. Opens the create form.
Search: Filters the list by name.
Output metric, PromQL, Interval, Folder, Updated by: The columns of the list, showing each rule’s configuration at a glance.

Create a derived metric

Click Create derived metric to open the create form. The form has four sections: output metric name, interval, folder, and the query builder.

Fields

Field Description Required

Field	Description	Required
Output metric name	The name of the new metric that the rulemanager will produce. Must be a valid Prometheus metric name (letters, digits, `_`, and `:`). The form auto-derives a sensible default once you pick a metric in the query builder — you can override it.	Yes
Interval	How often the rule re-evaluates. Accepts Prometheus duration strings (`1m`, `5m`, `1h`). The allowed range is `1m` to `1h`.	Yes (defaults to `1m`)
Folder	The folder this rule belongs to. Folders control who can edit and view derived metrics. If you don’t already have a suitable folder, create one first in Admin > Folders.	Yes
Query	The PromQL expression the rule evaluates. Built with the structured query builder (see Build the query).	Yes

Output metric name

The name of the new metric that the rulemanager will produce. Must be a valid Prometheus metric name (letters, digits, _, and :). The form auto-derives a sensible default once you pick a metric in the query builder — you can override it.

Yes

Interval

How often the rule re-evaluates. Accepts Prometheus duration strings (1m, 5m, 1h). The allowed range is 1m to 1h.

Yes (defaults to 1m)

Folder

The folder this rule belongs to. Folders control who can edit and view derived metrics. If you don’t already have a suitable folder, create one first in Admin > Folders.

Yes

Query

The PromQL expression the rule evaluates. Built with the structured query builder (see Build the query).

Yes

Build the query

The query step uses the exact same query builder as the Metrics Explorer — metric, filters, and operations, with the same Builder/Code toggle — so anything you already know about building a query there carries over directly. See The query builder for the full field reference.

Use + Query and + Formula the same way you would in the Explorer to build a multi-query expression — for example, two source queries combined with a formula like a / b for an error rate — before saving it as a derived metric.

The chart at the bottom of the form previews the result against live data as you build it. Confirm the preview looks right before saving.

Example: precompute total request rate per cluster

This example creates a derived metric that sums envoy_http_downstream_rq_total across all pods, grouped by cluster name.

Open Metrics > Derived metrics > Create derived metric.
In the query builder’s Metric field, type envoy_http_downstream_rq_total and pick it from the dropdown.
Replace the default Avg operation with Sum from the Aggregations category, grouped by kube_cluster_name.

The generated PromQL becomes sum by (kube_cluster_name)(envoy_http_downstream_rq_total).

The chart previews real data — typically a slowly-rising counter line per cluster.
Change the Output metric name to something descriptive, such as demo:envoy_rq_total:by_cluster. The hint below the field says "Query the result with $<your name>$ " — that is the exact name dashboards and alerts will use later.
Pick your folder from the Folder dropdown.
Click Create rule.

The rule starts evaluating immediately and produces its first sample at the next interval, and the new metric appears in the list.

Query the result

Once the rule has produced at least one sample (after one full interval), the new metric is queryable like any other.

In a dashboard or alert, reference it by the Output metric name you set:
```
demo:envoy_rq_total:by_cluster{kube_cluster_name="pisco"}
```
promql
Compose with other functions just like a regular metric:
```
rate(demo:envoy_rq_total:by_cluster[5m])
```
promql

To see the benefit, place the precomputed metric and the original expression side by side in the same dashboard: one panel running the heavy sum by (kube_cluster_name) (envoy_http_downstream_rq_total) live on every refresh, the other just reading the precomputed demo:envoy_rq_total:by_cluster series. Values agree within one evaluation interval (rules sample at fixed offsets, slightly before live query time).

Edit, pause, or delete a rule

From the derived metrics list, click into a rule’s row to open its details. From there, you can:

Edit — change the interval, folder, or query. Renaming the output metric is supported, but downstream dashboards or alerts referencing the old name need to be updated separately.
Pause — temporarily stop the rule from evaluating without losing its configuration. Existing samples remain queryable; no new samples are produced until you resume.
Resume — re-enable a paused rule. The first sample after resume comes at the next interval.
Delete — permanently remove the rule. The derived metric stops accumulating new samples; historical samples remain queryable until they age out of retention.

Define rules in Helm values

Derived metrics are Prometheus recording rules managed from the UI. You can also define recording rules in your Helm values file, so that they ship as part of your Kloudfuse configuration instead of being managed by whoever has UI access. Both kinds of rule run in the same rule manager in the metrics query service and produce the same kind of metric.

Rules defined in Helm values do not appear in the derived metrics list and cannot be edited, paused, or deleted from the UI. Metric Usage lists them as recording rules.

To add recording rules:

In your custom-values.yaml file, add one or more rule groups under query-service.rulemanager.rules.groups. Each rule needs a record name for the new metric and a PromQL expr:

query-service:
  rulemanager:
    rules:
      groups:
      - name: recording_rules
        rules:
        - record: application_availability_slo (1)
          expr: sum by (kube_cluster_name, kube_service) (increase(promhttp_metric_handler_requests_total{code=~"2.*"}[28d])) / sum by (kube_cluster_name, kube_service) (increase(promhttp_metric_handler_requests_total{code=~"2.*|5.*"}[28d]))
      - name: request_rates (2)
        interval: 1m (3)
        rules:
        - record: cluster:http_requests:rate5m
          expr: sum by (kube_cluster_name) (rate(promhttp_metric_handler_requests_total[5m]))
    ruleEvalSettings:
      EvalInterval: 30 (4)

yaml

1	`record`: the name of the new metric that holds the result of `expr`.
2	Add more groups as additional entries in the `groups` list. Group names must be unique.
3	`interval`: optional; how often to evaluate the rules in this group.
4	`EvalInterval`: how often, in seconds, to evaluate groups that do not set their own `interval`. The default is `30`.

Apply the change:

helm upgrade --install kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse \
  -n kfuse \
  --version <VERSION> \ (1)
  -f custom-values.yaml

bash

1	Replace `<VERSION>` with a valid Kloudfuse release value. See Release Notes for the latest release.

The groups use the standard Prometheus rule group format; see Prometheus recording rules.

Troubleshooting

The Create rule button stays disabled

Every required field must be filled. If the button is still disabled, check that the query builder produced a non-empty GENERATED PROMQL string — sometimes the structured builder needs the metric to be selected from the dropdown rather than typed and left in the input field.

The new metric does not show up in the dashboard panel picker

The picker reads from a periodically-refreshed metric metadata cache. Recording-rule outputs are only picked up after the cache refreshes (typically a few minutes after the first sample). In the meantime, the metric is still queryable — type the full output name in the metric box, or compose the panel URL directly. Once the cache refreshes, autocomplete starts working normally.

The new metric does not appear in the Metrics Cardinality Explorer

The cardinality explorer queries the 5-minute rollup table. Newly-created derived metrics live in the raw table first and only land in the rollup table after the rollup pipeline processes them — often after one or more rollup cycles. Use PromQL directly, or wait for the next rollup cycle.

The chart preview is empty

The preview runs the same expression against live data. If you see no data, double-check that the source metric is actually emitting samples in the current time range (top-right time picker, defaults to the last hour).