Upgrade Instructions

You can upgrade to Release 4.4.0 from Release 4.3.0 or Release 4.2.4. From an earlier release, upgrade to Release 4.2.4 first.

Log normalization (remap) turns each agent’s raw payload into a log’s message, timestamp, source, and labels. Release 4.3.0 changed how logs from several agents are normalized. Release 4.4.0 reverts those changes, so logs are normalized as in Release 4.2.4 again, apart from a few deliberate differences. What you need to check depends on the release you are upgrading from.

If you set retentionBytes in Size Metrics Topic Retention, confirm that the topic has it. Replace kfuse with your Kloudfuse namespace:

kubectl exec -n kfuse -c kafka kafka-kraft-broker-0 -- sh -c 'unset JMX_PORT; \
  /opt/bitnami/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name kf_metrics_topic --describe'

Copied!

The 4.4.0 upgrade may set short retention and segment settings on kf_metrics_rollup_topic so that Kafka frees its disk. A helm rollback below 4.4.0 does not remove them, and releases before 4.4.0 write rollups to this topic again, so rollups that are not read within minutes are lost.

kubectl exec -n kfuse -c kafka kafka-kraft-broker-0 -- sh -c 'unset JMX_PORT; \
  /opt/bitnami/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name kf_metrics_rollup_topic --describe'

Copied!

kubectl exec -n kfuse -c kafka kafka-kraft-broker-0 -- sh -c 'unset JMX_PORT; \
  /opt/bitnami/kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics --entity-name kf_metrics_rollup_topic \
  --alter --delete-config retention.bytes,retention.ms,segment.bytes,segment.ms'

Copied!

Create shaping rules with label filters only after every ingester is running Release 4.4.0.

Upgrade to Release 4.2.4 before upgrading to Release 4.3.0. Downgrading from 4.3.0 is not supported; the chart refuses to install an earlier version over it.

Both the existing UI and the new Kloudfuse UI now read their settings from global.uiConfig. The upgrade fails if your custom-values.yaml file still sets ui.config. Move each key and remove ui.config:

# Before
ui:
  config:
    <key>: <value>

# After
global:
  uiConfig:
    <key>: <value>

yamlCopied!

Archives are now written directly from the ingest stream through a new Kafka topic, kf_logs_archive_v2_topic. If archiving is enabled and your custom-values.yaml file overrides global.kafkaTopics, add this topic to the list, or the upgrade fails. The topic carries every ingested log line, so give it the same partition count as logs_ingest_topic:

global:
  kafkaTopics:
    - name: kf_logs_archive_v2_topic
      partitions: <same as logs_ingest_topic>
      replicationFactor: <same as logs_ingest_topic>
      minInSyncReplicas: <same as logs_ingest_topic>

yamlCopied!

The query engine servers restart during the upgrade (see Longer Pinot Server Pod Termination). Until they are back, expect partial query results, ingestion lag, and alerts that report errors or no data. Silence alerts for the duration of the upgrade.

With RBAC enabled, Release 4.3.0 requires the Admin role for creating, editing, starting, stopping, or deleting archives and for starting, pausing, resuming, or cancelling hydration jobs (see Admin-Only Archive and Hydration Management). Before upgrading, grant the Admin role to the users and service accounts that manage archives or run hydration jobs, so they keep that access after the upgrade. Grant it deliberately: the Admin role also gives unrestricted access to all telemetry and to user and policy administration (see Role Permissions). Listing and progress views are unaffected.

The Expensive Queries control-plane dashboard and the query counts and last-queried times on the Metrics Usage screen are built from performance logs, which are disabled by default in Release 4.3.0. Without them, the dashboard stays empty and Metrics Usage shows only which dashboards, alerts, and rules reference each metric. To enable them, set EnablePerfLogs in your custom-values.yaml file; performance logs also require RBAC and audit logs, which are enabled by default:

global:
  EnablePerfLogs: true

yamlCopied!

The new Kloudfuse UI is available at the /f path after the upgrade, with no configuration required. The existing UI remains the default experience; each user can switch individually, and the choice is remembered in the browser.

Pinot server pods now allow up to 21.5 minutes to drain in-flight queries and offload segments before terminating, instead of being force-killed after 30 seconds; if the graceful-termination segment commit (see Pinot Graceful Termination) is enabled, the commit timeout is added on top (5 more minutes by default). Rolling upgrades and restarts can therefore take longer than before; pods still terminate as soon as the drain completes.

Performance log entries from every query service now use the unified field set described in Performance Log Fields (see Unified Performance Logs). Each entry now carries the emitting service as its source; the previous source="kf-performance-log" filter matches no new entries, so use @perf_log="true" instead. Saved queries, scheduled views, or dashboards built on the previous fields — for example a duration parsed from a string rather than read from the integer duration_ms facet — also need to be updated.

Log normalization (remap) now runs in the ingest path by default (global.logs.remap.enabled: true) instead of in the log parser. The Logs Remap page is editable only when global.logs.remap.configDb.enabled is also true. See Log Source Normalization (Remap).

Log lines longer than 100 KB are now truncated at ingest. The previous limit was 1 MB.

Release 4.3.0 changes how log lines are split into searchable terms (see More Intuitive Term Search). The change is forward-only: it applies to log data written after the upgrade, while previously written data keeps the old behavior until it ages out of retention. During that window the same search can match new logs and miss older ones.

In Release 4.2.4, the logs, metrics, and trace transformers change from deployments to stateful sets, and they divide Kafka partitions among their pods using a new scheme that is not compatible with the previous one. The existing pods must be fully stopped before you upgrade, rather than replaced by a rolling update.

kubectl delete deployment -n <namespace> --cascade=foreground \
  logs-transformer metrics-transformer trace-transformer

Copied!

Release 4.2.2 configures GCP Pub/Sub log collection with a subscriptions list under global.gcpConfig.pubsub, replacing the previous single subscriptionId field. Update the pubsub section of your custom-values.yaml file to the new list format before upgrading:

global:
  gcpConfig:
    secretName: "kfuse-gcp-credentials"
    projectId: "<your-gcp-project-id>"
    pubsub:
      enabled: true
      subscriptions: (1)
        - projectId: "<your-gcp-project-id>" (2)
          subscriptionId: "kloudfuse-gcp-subscription"

yamlCopied!

Release 4.2.2 rejects an anchored parse pattern whose alias count does not match the number of extractions. Queries of this form were previously accepted and silently dropped extractions or left aliases empty. Saved searches, scheduled searches, and alerts using these patterns fail after upgrading and must be corrected.

Release 4.2.2 makes FuseQL query_logs the default logs tool in the MCP server. The legacy LogQL tools search_logs and query_log_metrics are no longer on the default tool surface. Clients that call them must either move to query_logs or request the legacy toolset explicitly with ?toolsets=logql.

Release 4.2.2 changes the response shape of the query_prometheus, search_prometheus_label_names, and search_prometheus_label_values tools. These tools now return parsed objects rather than JSON-encoded strings. Clients that decoded the previous string response must be updated. Other tool families are unchanged.

Release 4.2.1 requires a new Kafka topic. Before upgrading, add the following entry to global.kafkaTopics in your custom-values.yaml:

global:
  kafkaTopics:
    - name: kf_metrics_staging_topic
      partitions: 1
      replicationFactor: 1

yamlCopied!

Release 4.2.0 introduces the Metric Shaping Rules feature, which requires the kf_metrics_staging_topic Kafka topic. Before upgrading, ensure that this topic is included in your global.kafkaTopics section in the custom-values.yaml file:

        - name: kf_metrics_staging_topic
          partitions: <number-of-nodes>
          replicationFactor: 1

yamlCopied!

If you plan to enable the new archive writer (see Archive Writer Disabled by Default), ensure that the kf_logs_archive_v2_topic Kafka topic is included in your global.kafkaTopics section in the custom-values.yaml file:

        - name: kf_logs_archive_v2_topic
          partitions: <number-of-nodes>
          replicationFactor: 1

yamlCopied!

Starting in Release 4.2.0, the awsRegions list in Helm values defaults to an empty list. Customers using AWS CloudWatch metrics scraping must now explicitly specify the regions they want to scrape (opt in) — either globally under ingester.config.awsRegions, or per scraper in a specific awsRoleArns entry’s regions list. See Helm Values for details.

Query scheduling is enabled by default in Release 4.2.0. Administrators can configure per-user, per-group, and per-service-account query priorities — including the new blocked priority, which rejects matching queries instead of delaying or queueing them — from the admin UI. To disable query scheduling entirely, set the following in your custom-values.yaml:

global:
  queryScheduler:
    enabled: false

yamlCopied!

The new archive writer is disabled by default. It can be turned on per deployment when you are ready — contact Kloudfuse support to opt in. Enabling it requires the kf_logs_archive_v2_topic Kafka topic; see Archive Writer Kafka Topic (Conditional) in the pre-upgrade steps.

Scheduled searches are disabled by default and can be enabled on request — contact Kloudfuse support to enable the feature for your tenant.

If your configDB is hosted on an externally-managed Postgres instance (AWS RDS, GCP Cloud SQL, Azure Database for PostgreSQL) where the application user is not the default postgres, you must add an explicit events-query-service.config.rbacdb.user override to your custom-values.yaml. The events-query-service chart currently ships with a hardcoded rbacdb.user: postgres default that masks the global.configDB.username fallback, so without this override events-query-service connects to rbacdb as postgres and fails with no pg_hba.conf entry for host …​, user "postgres", database "rbacdb". Skip this step if global.configDB.username is postgres (the default).

events-query-service:
  config:
    rbacdb:
      user: <your-configdb-app-username>  # match global.configDB.username

yamlCopied!

Starting in 4.1.0, ingestion API keys and their optional additional labels can be managed from the Kloudfuse UI. Open Admin > Settings, then click Configure on the Auth key labels card. The legacy YAML configuration (kfuse-auth-ingest secret + ingester.config.authKeyAdditionalLabels) continues to work but is deprecated. YAML-sourced entries appear in the UI as read-only.

kubectl delete secret kfuse-auth-ingest -n <namespace> (1)

Copied!

1	Replace <namespace> with the namespace of your Kloudfuse deployment. Remove the ingester.config.authKeyAdditionalLabels block from custom-values.yaml and re-apply Helm values, then restart config-mgmt-service: kubectl rollout restart deploy/config-mgmt-service -n <namespace> (2) Copied!
2	Replace <namespace> with the namespace of your Kloudfuse deployment. For full migration steps and a placeholder helper script, see Migrate YAML-managed Auth Keys to the UI.

Release 4.0.1 upgrades the Envoy proxy to v1.36 with FIPS-compliant cryptography. The FIPS proxy rejects RSA private keys for TLS certificates delivered via SDS. All TLS secrets should be checked and any RSA certificates should be recreated as ECDSA.

After the upgrade, the Gateway resource has ECDSA annotations (cert-manager.io/private-key-algorithm: ECDSA). If your listener TLS certificate was RSA (identified in the pre-upgrade check), it must be deleted and reissued as ECDSA.

K8s deployment type for hydration-service has changed from Deployment to StatefulSet. Before upgrading to 4.0, delete the existing Deployment and ConfigMap.

Starting in 4.0.0, Kloudfuse supports Envoy Gateway as an alternative to NGINX Ingress. Existing deployments can optionally migrate using a zero-downtime 3-step migration process. See Configure Envoy Ingress for details.

Starting in 4.0.0, the awsNamespaces list in Helm values defaults to an empty list. Customers using AWS CloudWatch metrics scraping must explicitly list the namespaces they want to scrape. Before upgrading, add the desired namespaces under ingester.config.awsNamespaces in your Helm values override:

ingester:
  config:
    awsNamespaces:
      - "AWS/EC2"
      - "AWS/RDS"
      - "AWS/S3"
      - "AWS/Lambda"
      # Add other namespaces as needed

yamlCopied!

After the upgrade, wait for the setup-pinot job to complete, then perform a rollout restart of the Pinot STS (StatefulSet) to pick up the updated schema and table configuration.

Multi-resolution rollup increases memory usage in the metrics transformer. Review and increase the GOMEMLIMIT setting for the metrics transformer if needed, especially for high-cardinality deployments.

Metrics rollup is enabled by default in 3.5.3. Before upgrading, ensure that kf_metrics_rollup_topic is included in your Kafka topics list in the Helm values. Specify the same number of partitions as the existing kf_metrics_topic.

If RUM is enabled in your deployment, ensure that kf_rum_vitals_topic is included in your Kafka topics list in the Helm values. Specify the same number of partitions as the existing kf_rum_views_topic.

If you were using a non-default rollup interval before 3.5.3 (for example, 600s instead of the default 300s), set the legacy interval so that pre-3.5.3 rollup data is attributed to the correct resolution:

global:
  metrics:
    legacyRollupIntervalSecs: 600

yamlCopied!

After the upgrade, wait for the setup-pinot job to complete, then restart the Pinot STS (StatefulSet) to pick up the new rollup schema and table configuration. Existing metrics data is automatically compatible with multi-rollup resolution.

Review your Helm values if you use custom security configurations. All services now run as non-root users by default and support configurable service accounts and security context. Existing deployments without custom security settings are unaffected.

Scheduled views have been redesigned. You must recreate your scheduled views after upgrading. Data continuity from the previous implementation cannot be guaranteed.

Starting with 3.5.2, all Kloudfuse container images and Helm charts are signed. You can optionally verify image signatures before deployment.

If you use folder-based organization for dashboards and alerts, review folder permissions after upgrade to ensure appropriate access levels are configured.

After upgrading to 3.5.2, you must restart the logs-parser to ensure proper functionality.