Metrics rollup processes

The following diagram illustrates how Kloudfuse handles metrics, from live stream processing, to queries from dashboards and alerts. It describes the stages of ingestion, processing, calculation, storage, and query processing.

Diagram of metrics rollup: the top band shows ingestion — (1) sources such as agents, cloud sources, and remote write send metrics to the Ingester, which (2) writes them to the Kafka topic kf_metrics_topic; (3) raw metrics go straight to Pinot, while the Metrics Transformer aggregates each series per 5-minute and 30-minute window and writes each resolution to its own topic, kf_metrics_rollup_300_topic and kf_metrics_rollup_1800_topic; (4) the Metrics Decoder writes raw samples to the kf_metrics table, and the Rollup Decoder writes sum, count, min, max, counter, first, and first_ts to the kf_metrics_rollup table, also still reading the legacy kf_metrics_rollup_topic, which is no longer written after 4.4.0. The bottom band shows queries — (1) the Kloudfuse UI sends a query to the Query Service, which (2) selects a resolution from the query’s step size and time range, (3) reads kf_metrics for short ranges and small steps and kf_metrics_rollup for longer ranges, splitting the query when only part of the range has rollup data, and (4) returns the merged result

Metrics ingestion, processing, calculation, storage, and queries

Metrics ingestion, processing, and storage

Refer to the upper part of the Metrics ingestion, processing, calculation, storage, and queries diagram that illustrates the rollup workflow. The numbers in light blue circles correspond to these steps:

Kloudfuse gets time series data from your environment, either through agents or from cloud sources.
The Ingester Service pre-processes the data stream, and routes it to Kafka as kf_metrics_topic.
The same data stream follows two parallel paths:
- Raw metrics
- Rollup metrics
Kafka forwards the kf_metrics_topic directly to Pinot.
The rollup path reads kf_metrics_topic and computes rollup metrics:
Kafka delivers kf_metrics_topic to the Metrics Transformer.

The Metrics Transformer calculates aggregations and markers for each rollup resolution (5 minutes and 30 minutes), and sends each resolution back to Kafka on its own topic: kf_metrics_rollup_300_topic and kf_metrics_rollup_1800_topic.

Kafka forwards both rollup topics to Pinot.

Pinot handles the topics in the following manner:
- Raw metrics
- Rollup metrics
The Metrics Decoder receives kf_metrics_topic, performs necessary calculations, and writes it to table kf_metrics.

The table columns are: name (of metric), timestamp, labels, value, and le.

The Metrics Rollup Decoder receives the rollup topics, performs necessary calculations and aggregations, and writes it to the table kf_metrics_rollup.

The table columns are name (of metric), timestamp, labels, sum, count, min, max, counter, first, first_ts, and le.

The Metrics Transformer calculates sum, count, min, and max from the raw samples in each rollup window, before they reach Pinot.

Kloudfuse uses both counter (last counter value that accounts for resets within the rollup window), first (first value encounter in the bucket), and first_ts (timestamp of first) to ensure data integrity.

Metrics queries

Refer to the lower part of the diagram that illustrates the query workflow.

The numbers in the dark blue circles correspond to these steps:

Kloudfuse gets a query request from a user interface.

This may be triggered by starting the Metrics interface, loading dashboards, changing and reloading dashboards and reports, changing the time picker values, and so on.
The Query Service selects the most appropriate rollup resolution based on the query’s step size and time range. For queries with a small step size or short time range, it reads from the raw table. For longer time ranges, it selects a coarser rollup resolution to reduce the amount of data scanned.
The Query Service receives results from:

Raw metrics

Table kf_metrics

Rollup metrics

Table kf_metrics_rollup

If the query spans a time range where only part of the data has rollup coverage, the query service automatically splits the query — using the rollup table for the newer portion and the raw table for the older portion.
The Query Service combines the results and returns them to the requesting UI.