Implementing Retention Policies

Kloudfuse has default and additional custom policies for various data streams. See Retention Architecture for why these policies are tied to Kafka partitioning the way they are.

Stream-level Retention

By default, the Kloudfuse stack installs with stream-level retention policies for every stream in global.retentionPolicy: logs, metrics, events, traces, rum, and logs_views. You configure these in the global.retentionPolicy section of your custom-values.yaml.

For logs_views, the policy below is only a fallback: every Scheduled View row is written with its own expiry, 14 days by default. See Interval and retention.

Metric rollups can keep a different retention from raw metrics, for all rollups or for each rollup resolution. See Configure metrics rollup.

Here are the chart’s own defaults:

How to Set Default Retention Policies
global:
  retentionPolicy:
    expireRoundUp: 10800000 # 3 hours, in ms
    logs:
      default:
        retentionTimeValue: 7
        retentionTimeUnit: DAYS
    metrics:
      default:
        retentionTimeValue: 30
        retentionTimeUnit: DAYS
    events:
      default:
        retentionTimeValue: 30
        retentionTimeUnit: DAYS
    traces:
      default:
        retentionTimeValue: 7
        retentionTimeUnit: DAYS
    rum:
      default:
        retentionTimeValue: 7
        retentionTimeUnit: DAYS
    logs_views:
      default:
        retentionTimeValue: 365
        retentionTimeUnit: DAYS
yaml

retentionTimeUnit and retentionTimeRollupUnit accept SECONDS, MINUTES, HOURS, DAYS, WEEKS, MONTHS, and YEARS, in default and custom policies. WEEKS, MONTHS, and YEARS count as 7, 30, and 365 days, so 1 MONTHS is always 30 days, not a calendar month. If a unit is not recognized, Kloudfuse ingestion services fail to start.

Releases before 4.4.0 support only DAYS, HOURS, MINUTES, and SECONDS. They read any other unit as milliseconds.

expireRoundUp is required at the top of retentionPolicy — every stream shares it. See The expiry column for what it’s actually used for.

Record-level Retention

Kloudfuse also supports configuring retention at the level of individual records, by adding custom policies alongside a stream’s default one. This is available for logs, metrics, events, and traces.

Be aware that there are performance and operational costs to each additional policy: every custom policy you add means dividing a topic’s partitions into one more group (One topic, one contiguous partition group per policy), and you must keep that topic’s partition count divisible by the new total. We recommend no more than 3 additional policies per stream.

Configure Additional Retention Policies

Add a custom list alongside a stream’s default entry, under global.retentionPolicy.<stream>. Each custom policy needs retentionTimeValue, retentionTimeUnit, and a name — the name is what a relabel rule targets to assign a record to that policy.

Always add new custom policies to the end of the list. Reordering or deleting an existing policy changes every later policy’s installation index — and that index is what determines which Kafka partition group it owns (One topic, one contiguous partition group per policy). Reordering silently remaps old data to a different policy’s partition group.

The following example sets default and two custom policies each for metrics and logs:

How to Set Default and Custom Retention Policies
global:
  retentionPolicy:
    expireRoundUp: 10800000 # 3 hours. Required.
    metrics:
      default:
        retentionTimeValue: 5
        retentionTimeUnit: DAYS
      custom:
        - retentionTimeValue: 7
          retentionTimeUnit: DAYS
          name: prod
        - retentionTimeValue: 14
          retentionTimeUnit: DAYS
          name: staging
    logs:
      default:
        retentionTimeValue: 14
        retentionTimeUnit: DAYS
      custom:
        - retentionTimeValue: 30
          retentionTimeUnit: DAYS
          name: prod
        - retentionTimeValue: 90
          retentionTimeUnit: DAYS
          name: staging
yaml

Configure Kafka Partitions

Every custom policy you add means re-sizing that stream’s Kafka topic — see Sizing Kafka topics and PVCs for the full reasoning and worked examples. The short version: the chart’s topic-creation job rejects an install if a covered topic’s partitions isn’t evenly divisible by 1 + len(custom policies) for that topic’s stream.

Only some topics are covered by that automatic check — the job maps a fixed set of topic names to a stream and validates only those:

Topic Stream validated

kf_metrics_topic

metrics

kf_metrics_rollup_topic

metrics

kf_metrics_rollup_300_topic, kf_metrics_rollup_1800_topic

metrics

kf_events_topic

events

kf_traces_topic

traces

kf_traces_errors_topic

traces

kf_logs_topic

logs

The RUM topics (kf_rum_*_topic) are not checked. Custom retention policies aren’t supported for RUM, so RUM data always uses the rum default policy and its topics need no partition changes.

How to Configure Kafka Partitions
global:
  kafkaTopics:
    - name: kf_metrics_topic
      partitions: 3   # 1 default + 2 custom policies
      replicationFactor: 1
      minInSyncReplicas: 1
yaml

kf_logs_topic has one additional field, hydrationPartitions, that sits outside this divisibility rule entirely — those partitions are reserved for replaying archived logs (Hydrate From Archive) and are never divided among retention policies. Only partitions needs to be divisible by the logs stream’s policy count:

global:
  kafkaTopics:
    - name: kf_logs_topic
      partitions: 2            # divided among logs' retention policies
      hydrationPartitions: 1   # separate, reserved for hydration replay
      replicationFactor: 1
      minInSyncReplicas: 1
yaml

If global.archive.enabled is true and you don’t set hydrationPartitions explicitly, it defaults to global.numNodes rather than to a fixed small number — size `kf_logs_topic’s total partition count (and its PVC) with that in mind.

Configure Relabel Rules

Metrics, Events, and Traces

Kloudfuse uses relabel rules to attach a retention policy to each record. A record that doesn’t match any custom policy’s relabel rule falls back to the stream’s default policy. These rules live under ingester.config in custom-values.yaml, with one relabelConfigs list per stream:

Stream Helm values path

Metrics

ingester.config.aggregator.metric.relabelConfigs

Events

ingester.config.events.relabelConfigs

Traces

ingester.config.traces.relabelConfigs
Custom retention policies aren’t supported for RUM; RUM data always uses the stream’s default policy. See the retention action in How relabel rules work.

Each entry follows the Prometheus relabel_config format: source_labels (a list), regex, and for the retention action, a retention_policy field naming the target policy (matching a name you defined under Configure Additional Retention Policies). Rules are evaluated in order; if a record matches more than one, the last match wins.

How to Configure Metrics, Events, and Traces Streams
ingester:
  config:
    aggregator:
      metric:
        relabelConfigs:
          - source_labels:
              - env
            regex: "prod"
            action: retention
            retention_policy: prod
          - source_labels:
              - __name__
            regex: "test_app(.*)"
            action: retention
            retention_policy: staging
yaml

In this example, the prod policy applies to metrics with an env label equal to prod; the staging policy applies to any metric whose name starts with test_app. The same shape applies under ingester.config.events.relabelConfigs and ingester.config.traces.relabelConfigs for their streams.

Logs Stream

Logs use a different mechanism entirely, because they reach Kafka through logs-parser rather than the ingester. Instead of an action: retention relabel entry, set the reserved __kf_retention label to the policy name:

How to Configure Logs Streams
logs-parser:
  kf_parsing_config:
    config: |-
      - relabel:
          args:
            - action: "replace"
            - sourceLabels: "#source"
            - targetLabel: "__kf_retention"
            - regex: "^.*-stag-.*$"
            - replacement: "staging"
yaml

This example assigns the staging policy to any log whose source label contains -stag-. See Log Parsing Configuration for the full relabel reference, and Logs-parser: logs for why this path looks different from the other streams.

Next steps