Configure metrics rollup

Starting with version 3.5.3, metrics rollup is enabled by default. To disable it, set rollupEnabled to false in your custom-values.yaml file:

Starting with version 4.4.0, Kloudfuse rolls up metrics at two fixed resolutions: 5 minutes and 30 minutes. When rollup is enabled, both resolutions are always produced, and they cannot be changed.

Each resolution is written to its own Kafka topic, and the kf_metrics_rollup table consumes both:

Both topics are required when rollup is enabled. The chart’s default topic list includes them, so you only need to add them if your custom-values.yaml file overrides global.kafkaTopics. In that case, add them to your existing list, with the same settings as your existing kf_metrics_rollup_topic: the same partition count and, if that entry sets them, the same replicationFactor and minInSyncReplicas. If your kf_metrics_rollup_topic entry does not set replicationFactor or minInSyncReplicas, leave them out of the new entries as well, so they get the same defaults. Keep kf_metrics_rollup_topic listed as well; nothing writes to it after the upgrade, but the kf_metrics_rollup table still reads it. If a required topic is missing, the Helm upgrade fails and prints the entries to add.

The rollupIntervalSecs and rollupResolutionFactors settings from earlier versions are no longer supported. Remove them from your custom-values.yaml file; the Helm upgrade fails if either is set.

The query service automatically selects the most appropriate resolution based on the query’s step size and time range. For shorter time ranges or smaller step sizes, raw metrics are used. For longer time ranges, a coarser rollup resolution is selected to improve query performance.

By default, rolled up metrics have the same retention policy as raw metrics. To use a different retention period for all rollups, set retentionTimeRollupValue and retentionTimeRollupUnit on the retention policy in the global.retentionPolicy.metrics section of your custom-values.yaml file.

Starting with version 4.4.0, you can also give each rollup resolution its own retention with rollupRetentionByResolution, keyed by the resolution in seconds: "300" for 5 minutes and "1800" for 30 minutes. A resolution that is not listed keeps the rollup retention. For example, the following policy keeps raw metrics for 30 days, 5-minute rollups for 90 days, and 30-minute rollups for 365 days:

You can set retentionTimeRollupValue, retentionTimeRollupUnit, and rollupRetentionByResolution on the default policy and on each custom policy. Supported units are SECONDS, MINUTES, HOURS, DAYS, WEEKS, MONTHS, and YEARS; WEEKS, MONTHS, and YEARS count as 7, 30, and 365 days. A unit that is not recognized in retentionTimeRollupUnit stops ingestion services at startup; in rollupRetentionByResolution, the Helm upgrade fails. See Supported retention units. A rollupRetentionByResolution change applies only to rollups written after the change; rollups that are already stored keep the retention they were written with.

When some older data has no corresponding rolled up data, the query service automatically handles this by splitting the query. It uses the raw kf_metrics table for the older portion and the rolled up kf_metrics_rollup table for the newer portion, then merges the results before returning them.