Fixed Issues

FuseQL logs alerts now work across the two UIs. An alert saved in the new Kloudfuse UI can be edited and saved in the existing UI, where saving previously failed; an alert last saved in the new UI on Release 4.3.0 must be saved once in the new UI after the upgrade. An alert created in the existing UI now opens with its query in the new UI instead of an empty one.

In the existing UI, the policy health findings list on the contact points page is no longer clipped, and can be scrolled.

Saving an alert rule in the new Kloudfuse UI no longer changes how it evaluates. Previously, saving could:

Importing alert rules in the new Kloudfuse UI whose queries have no datasource block now assigns each query the correct datasource in this environment. Previously these rules kept the source environment’s datasource and never evaluated. Re-import any rules imported this way: in the review step, choose Replace for each rule, and either clear the option to import it paused or resume the rules after the import.

The new Kloudfuse UI no longer lets you edit or copy a LogQL alert. Previously, saving a LogQL alert there, even to change only its threshold, converted it to FuseQL and the rule stopped evaluating. Check any LogQL alerts that were edited in the new UI.

Alerts created or edited in the new Kloudfuse UI now store the query or formula selected in the condition step. Previously they always stored the first query, so a condition on another query or on a formula evaluated the wrong query. Save affected rules again.

Pausing or resuming an alert in the new Kloudfuse UI now also pauses or resumes its warning threshold. Previously the warning threshold kept firing while the alert showed as paused. Pause such alerts again.

Renaming a contact point in the new Kloudfuse UI now moves its alert rules and routing to the new name; previously rules kept the old name and their notifications were not delivered. A destination added to an existing contact point is also no longer removed right after it is created. Check contact points that were renamed or extended in the new UI.

Change alerts that group by labels were saved in a form that never evaluated when created or edited in the new Kloudfuse UI, including rules first created in the existing UI. Change alerts on a rate query with the default $__rate_interval range, or a subquery range, had the same problem. They are now saved correctly; save affected rules again after the upgrade.

Saving a metrics threshold or outlier alert in the new Kloudfuse UI no longer drops its Evaluate as Zero no-data setting, which changed how the rule evaluated, and the setting can now be chosen there. If an alert had Evaluate as Zero turned on before it was saved in the new UI, turn it on again.

The Kloudfuse Source link in alert notifications now opens the alert details page filtered to the firing instance when a label value contains &, +, %, or =. Previously such values were cut short or changed, so the details page showed no matching instance. Links in existing rules are updated the next time the rule is saved in either UI.

In the new Kloudfuse UI, if a suppress schedule failed to load while you edited it, the page showed a blank schedule that could be saved over the real one, which muted every contact point that used the schedule at all times. The page now shows the error instead. A schedule that was already overwritten this way stays active at all times: check suppress schedules edited in the new UI, and set their recurrence again.

Dashboards and panels whose titles contain characters outside the Latin-1 range, such as an em dash or an emoji, no longer fail to load their panel queries. In the new Kloudfuse UI this affected stat, gauge, bar gauge, and table panels; in the existing UI it affected every panel on the dashboard.

Control Plane pages, and the Nodes and Volumes tabs of a service in the existing UI, now open the dashboard that ships with Kloudfuse. Previously, when a dashboard with the same title existed in another folder, they could open that dashboard instead.

Panels saved from the new Kloudfuse UI now render in the existing UI. Previously, metric time series panels edited in the new UI showed No data or a blank page in the existing UI, FuseQL log panels failed with a parse error, and charts added from the Logs, Traces, Events, or RUM explorers showed Failed to load panel. The existing UI now also displays panels that were already saved this way.

Dashboard variables whose query is stored in object form, such as in dashboards imported from Grafana, now open correctly in the new Kloudfuse UI’s variable editor. Previously editing one there flattened its query, which also broke the variable in the existing UI. Check variables that were edited in the new UI.

Metric Usage now records queries from the Metrics Explorer, from dashboards in either UI, and from recording, alerting, and SLO rule evaluations. Metrics that were queried only in these ways no longer appear as not queried.

Label, label value, series, and metric name lookups over long time ranges no longer exhaust memory on the storage servers. By default, these lookups now search at most the last 30 days and return at most 50,000 results; longer ranges return the most recent 30 days, with a warning. To change the limits, set query-service.config.Metric.MaxMetaQueryLookbackDays (0 removes the time limit) and query-service.config.Metric.MetaQueryResultLimit.

Metric queries and alert rule evaluations that end near the current time and are served from rollups could miss up to several hours of the newest data, or show a dip at the right edge of graphs. They now read the newest part of the range from a finer rollup or from raw data.

Queries that compute increase on histogram buckets and group by le, such as sum by (instance, le) (increase(http_request_duration_seconds_bucket[5m])), no longer show an occasional one-step dip on the lowest buckets. While the upgrade is rolling out, these queries can return errors until all pods run Release 4.4.0.

CloudWatch HTTPCode_ELB_4XX counts for classic Elastic Load Balancing load balancers are now stored as aws_elb_httpcode_elb_4xx instead of aws_elb_httpcode_elb_3xx, as described in AWS CloudWatch Metric Name Mapping. Names with a statistic suffix change the same way; for example, aws_elb_httpcode_elb_3xx_sum becomes aws_elb_httpcode_elb_4xx_sum. Older data keeps the old name, so update dashboards and alerts to the new name and, while older data is retained, combine both, for example sum(aws_elb_httpcode_elb_4xx{<filters>} or aws_elb_httpcode_elb_3xx{<filters>}).

When a metrics retention policy sets a longer rollup retention, rollups of classic histograms, and of metrics generated from logs and traces, expired after the shorter raw retention. They now keep the rollup retention.

The Top K by label control in the new Kloudfuse UI’s metrics query builder now returns K series, grouped by the selected labels. Previously it returned the top K series within each label group, often many more than K. Dashboard panels and alerts built with it keep the old query until they are opened and saved again in the new UI.

Lookup and lookup-contains queries on logs no longer return wrong matches or fail when the field name and the table column name differ.

Queries that use a lookup table no longer use large amounts of memory when a column of the table contains very long values. Lookup tables created before this release pick up the fix the next time they are refreshed.

Release 4.3.0 changed the source, agent, label, and level values of incoming logs for several agents. Logs are again normalized as in Release 4.2.4: OpenTelemetry logs no longer get a source of unknown and their agent is otlp again, remap rules without conditions apply to every agent, every matching rule applies, and archive rules that match on the log message take effect. A few deliberate differences from Release 4.2.4 remain; see Review Log Normalization Changes.

FuseQL aggregations and filters that refer to built-in fields by their kf_ names, such as kf_ts, kf_level, and kf_source, now return results instead of NaN, empty values, or a count of zero.

A FuseQL query that ended in | limit N after an aggregation, such as * | count by (level) | limit 2, could return more than N groups, some of them with counts that were too low. It now returns N groups with their full counts.

Sorting a fingerprint list by a facet that it is grouped on no longer fails with an internal error.

A logs query that read facet values from data still being ingested could occasionally show a neighboring log’s value for a facet, or fail with an internal error. These queries now return the correct value. Count queries that filter on two or more facets could also return wrong or empty counts on older log data; they now return correct counts.

With the default settings in 4.3.0, a scheduled view could stop receiving new results after enough view data accumulated. Scheduled views now continue to update.

Tag lookups for Redshift, Glue, Athena, and SES resources in the AWS EU ISO regions now succeed; previously they used the wrong AWS partition and returned no tags.

Performance and audit log lines are now parsed correctly, so their fields are faceted properly, and the request line is no longer blank for alert rule, scheduled search, and view evaluations.

Profiles produced by advance functions are no longer occasionally rejected by the Pyroscope server because of invalid characters.

Storage servers now load their data before they rejoin the cluster after a restart, which avoids long startup delays.

When management of the data store moved to another node while a lot of work was pending, it could stall, sometimes for hours, before it resumed managing data. It now resumes promptly.

When the data store’s saved read position in an ingest stream had expired, it resumed from the newest data and silently skipped the data still retained. It now resumes from the oldest retained data.

Archive writer pods are now spread evenly across nodes as their topology spread constraints intend.

WEEKS, MONTHS, and YEARS are now supported, as 7, 30, and 365 days, and retention runs for these streams. Data written before the upgrade was marked to expire almost at once, so retention deleted almost all of it. A unit that is not recognized now stops ingestion services at startup instead of being read as milliseconds. See Supported retention units.

A paused shaping rule no longer keeps diverting its metric away from normal ingestion while the shaped metric is not being produced.

Retention, drop, and keep rules now match labels attached through an API key.

A failed cloud resource lookup no longer crashes ingestion, and an error while reading New Relic entity labels no longer causes New Relic metric requests to fail.

On clusters that report many Kubernetes objects, updates to the stored object state could block one another for tens of seconds. Meanwhile, some metric and log ingestion requests were rejected, and the database volume holding the object state could fill up.

Setting resources under pinot.broker or pinot.server in your custom-values.yaml file no longer produces an invalid manifest; the values are now applied.

On busy clusters, the periodic move of older data to the long-term storage servers could keep retrying for hours without moving any data. It now completes.

When hydration log parsing ran with more than one replica, only the first replica did any work. All replicas now share the hydration workload.

An archive created in the UI or through the API first read through all retained log data before it archived its first logs, which could take several minutes. A new archive now starts reading from the time it was created.

An archive created, started, stopped, or deleted in the UI or through the API took effect only on the archive writer pod that handled the request until the pods restarted. It now takes effect on every archive writer pod.

A stuck archive loop is now detected and restarted.

Scrub jobs now delete matching data from long-term metric rollups (metrics jobs), error spans (APM jobs), and vitals (RUM jobs). Previously this data was left in place while the job still reported success. A job now reports a failure if it cannot scrub some of the matching data. In the existing UI, the RUM scrub preview now also counts errors and vitals, so you can scrub a filter that matches only those.

After an archive was deleted, the archive dashboard kept showing it with growing lag for up to a week, and an archive re-created with the same name resumed from where the old one had stopped. Deleting an archive now removes its progress, including for archives deleted before the upgrade.

After a restart, a query server of the data store could stay out of service for several minutes. It now rejoins the cluster quickly, including on the first restart after the upgrade.

Queries that time out, or are stopped because they use too much memory, now stop promptly and are reported as timed out or stopped, instead of as a general query error.

The dropped-requests count on the control-plane ingest dashboards included Datadog agent payloads that Kloudfuse accepts but intentionally does not store, so some routes appeared to drop most of their requests. The dropped count now includes only rejected requests and has a reason label; accepted payloads that are not stored are counted separately as ignored.

Resource requests and limits set under kfuse-rum-vector.resources in your custom-values.yaml file were ignored. They are now applied to the RUM collector pods.

A signed-in user could add certain internal request headers to a query API call and have the query evaluated with another user’s access, or with access to all data. Kloudfuse now removes these headers at the ingress, and they can no longer replace the access that was resolved for the caller.

A user with no log access policy, when the default log access is none, could use logs live tail and create, view, or delete log-derived metrics without any restriction on the data. These requests are now denied, and access-policy denials return 403 Forbidden instead of 401 Unauthorized.

The folder-mappings endpoint under /rbac is no longer reachable from outside the cluster.

The Java debug and management ports on the data store pods accepted unauthenticated connections from anything that could reach the pod’s IP address. These ports now listen only inside the pod. Access through kubectl port-forward still works.

Kloudfuse MCP server tools are easier for agents to call:

Downloaded fingerprint results no longer have an empty pattern column.

The customer-scoping filter in the logs view now lists every customer value, instead of only All and None on some deployments.

Sorting the logs list by a label and paging to the next page now works on every time window; it previously failed on short windows and omitted the newest data on longer ones.

Filtering from a log table cell whose column came from a label projection (for example | @kube_container_name as kube_container_name) returned no results and left a duplicate filter chip. The filter now matches the label correctly.

A LogQL range query with an offset modifier no longer shifts its window by twice the offset.

A FuseQL if() whose condition uses a term or pattern matcher (such as matches) no longer fails with an unsupported-function error.

Event count and time-series aggregations in the Events explorer were capped at 9,999 rows; they now follow the platform default limit. See Events API Reference for the current default limit on each Events API operation.

Creating or editing a notification template failed with an error, while listing and deleting still worked — so the screen looked healthy until you pressed save. Both write paths now work.

Importing alerts reported "0 alerts imported" for rules created in the built-in alert rule editor rather than the Kloudfuse alert form — including Kloudfuse-generated SLO and ASM rules — because the importer could not resolve their data source. Such rules now import correctly.

We fixed notification policy handling for alerts with multiple contact points, which could send duplicate notifications, fail to save with a permission error, and block users with folder edit permissions from editing the affected alerts.

Alert rules whose contact point is set through simplified routing no longer lose that setting when any rule in the same group is created, edited, or deleted, and the contact point is now displayed.

The alert details view no longer crashes on alerts with an unrecognized type — for example a rule that reuses one of the annotation keys Kloudfuse reserves; see Alert Annotations.

Editing an alert whose query uses advanced PromQL binary modifiers now falls back to the raw PromQL editor instead of corrupting the query.

The step field in alert query settings is now labeled preview-only: it affects only the preview chart, and the evaluation step is chosen automatically.

One malformed alert rule no longer crashes the entire alerts list, and rules with an unreadable annotation are now skipped at import with the reason shown.

An anomaly or forecast alert whose query matches no rows now evaluates as no data instead of failing on every evaluation.

The panel menu is no longer hidden behind the loading overlay while a panel refreshes.

The infrastructure view on the APM service page now shows metrics only from the pods backing that service, instead of including unrelated pods.

Editing a metric shaping rule now shows the saved source metric and lets you add labels by typing them, a folder created from the form is selected automatically, and the drop preview no longer shows misleading zeros in edit mode.

More query failures now return their actual error message and remediation text instead of a generic "Request failed". See Standardized Query API Error Responses and API Response Codes.

We fixed an intermittent crash in the query engine caused by a race while loading native libraries, by loading them at startup.

Fixed an issue where log queries that chained a parse, JSON, or advanced-function stage (for example cat | parse, | json) could hang or fail instead of returning results.

Fixed an issue where some log aggregation results were missing rows because the underlying identifier was not included in the query.

Fixed an issue where scrolling through large log result sets could skip or duplicate entries when a page boundary fell between result windows.

Fixed an issue where the time filter on scheduled-view log queries was not always honored.

Fixed an issue where an invalid timestamp could cause log date formatting to error instead of degrading gracefully.

Logs Analytics now shows the "no data" state only after a query has finished running, rather than while results are still loading.

Fixed several issues in the logs facet explorer so facet values load and filter correctly.

Fixed an issue where view names were incorrectly quoted in scheduled-view log links, so the links now open the expected logs.

Fixed an issue where histogram_quantile results could be inaccurate when a classic histogram’s buckets straggled across storage segments, so percentile queries now reconcile bucket boundaries correctly.

Fixed an issue where functions that do not take a vector selector could be misclassified as time aggregations, producing incorrect query results.

Fixed an issue where Prometheus range values were not always parsed as numbers.

The UI now displays - for a NaN value instead of a misleading 0.00ns.

Fixed an issue where the PromQL builder attempted to parse unsupported $__ template variables, causing query errors on affected dashboard panels.

Chart warning tooltips now appear instantly on hover so panel warnings are easier to see.

Fixed an issue where pausing or resuming an alert could show both a success and an error notification at the same time.

Fixed an issue where some series evaluations were missing from the no-data section of an alert.

Fixed a crash that occurred when disabling a selected query on a non-threshold log alert.

Fixed an issue where rule and folder names containing special characters were not escaped correctly in alert matchers.

Fixed a visual flash that occurred when toggling the detection method while editing a log alert.

Fixed an issue where a NOT filter with no operand produced malformed trace queries.

Fixed an issue where backslashes in span_name facet filter values were not escaped, causing incorrect filtering.

Span logs are now correlated by trace ID across Datadog, OpenTelemetry, and AWS X-Ray traces.

Fixed an issue where session queries that did not match the configured session fields could return an error instead of an empty result.

Fixed the search bar on the RUM performance page.

Fixed an issue where some RUM facet values were hard-coded or incorrectly capitalized.

Fixed an issue where the event filter dropdown did not load all available facets and labels.

Fixed an issue with cross-account metric ingestion.

Fixed an issue with region handling during metric scraping.

Fixed the precedence used when mapping ingestion auth keys to their additional labels.

Alerts and dashboards now show the username of the user who last updated them.

Fixed an issue in the MCP OAuth JWT path so the Authorization header is forwarded correctly, matching service-account behavior.

Fixed an issue where the key exists log filter could be applied with an undefined source, causing queries to return incorrect results.

Fixed an issue where the logs chart could remain in a loading state when a streaming query completed with no matching data.

Fixed an issue where Logs Analytics did not surface an error condition when a query failed.

Logs Analytics now surfaces backend warnings to the user when a query is partially served.

Fixed an issue where label-based authorization filters could be applied to log view queries that did not contain those labels, surfacing an UnknownColumnError to users with RBAC enabled.

Fixed an issue where Logs panels added to dashboards could not be opened for editing because of an export-shape mismatch.

Fixed an issue where a search term wrapped in single quotes could be dropped from the resulting log query.

Fixed an issue with advanced function operands, and incompatible visualization tabs are now disabled when the active query cannot produce them.

Fixed an issue where multi-value fields emitted by FuseQL queries did not align with the previous log query format.

Fixed an issue where object-shaped facet values could not be located in log events when looking up facet values from a row.

The Metrics query service now returns an appropriate HTTP error code when the underlying engine returns an error, making client retry behavior more predictable.

Fixed an issue where dashboards did not preserve a saved default value for template variables.

Fixed an expandable-cell layout regression and made the fingerprint cell self-manage its expanded state.

Fixed an issue where the recovery threshold operator was inverted relative to the alert threshold operator.

Fixed an issue where the session and view timing in the RUM Tab link area did not have consistent padding.

Added a new API for retrieving span info that returns the correct values used by the APM trace detail panel.

Added a refresh button on the folder permissions panel and a note explaining the propagation delay after a permission change.

Fixed an issue where the audit context did not load correctly, causing audit log details to be missing.

Audit log entries now include additional context to make user-management actions easier to attribute.

Fixed a potential nil pointer dereference in the ingester error path.

Set explicit HTTP server timeouts on the ingester to prevent idle connections from accumulating over time.

Timed-out segment uploads are now cancelled cleanly so they do not consume broker resources after the deadline.

The segment creation metadata CRC is now re-stamped before the segment is tarred for upload, preventing CRC mismatches on the receiving side.

The segmentBuildFailure meter is now exposed via the JMX Prometheus exporter so segment-build failures can be alerted on.

Fixed a crash in the log query service when running certain types of queries.

Fixed a crash when previewing lookup table results.

Fixed the concat() function in FuseQL to correctly concatenate string values.

Fixed an issue where anomaly forecast bounds were not displayed in the Metrics Explorer when querying a metric without an aggregation function.

Fixed an issue where the Metrics Explorer did not display query statistics when using formula-only queries.

Fixed an issue where the contact points configuration table did not scroll properly when a contact point had many email recipients.

Fixed an issue where the trace heatmap did not render.

Fixed an issue where SLO create, edit, and delete operations did not enforce folder-level permissions.

Fixed an issue where users could move an SLO from a folder they did not have access to. Both source and destination folder permissions are now checked.

Fixed an issue where editing an SLO failed when RBAC policies were enabled.

Fixed an issue where imported log-based alerts were not editable because the datasource UID was not matched correctly during import.

Fixed an issue where changing the dashboard time range triggered a full reload of all template variables, causing slow dashboard performance. Now only RUM template variables are re-fetched on time range changes; Prometheus and Loki template variables are preserved without re-querying.

Fixed an issue where stat panels could display an invalid number of decimal places.

Fixed accuracy issues with metrics rollup queries, including overcounting on counter resets and incorrect extrapolation.

Fixed an issue where the default series limit prevented aggregation functions (count, sum, avg) from working on high-cardinality metrics.

Fixed an issue where deleting saved queries did not enforce folder-level permissions, allowing users to delete saved queries from folders they did not have access to.