Azure Log Integration

Kloudfuse ingests Azure logs by routing them through an Azure Event Hub and a custom Azure Function App that forwards records to the Kloudfuse HTTP ingestion endpoint.

For an overview of how this fits into the broader Azure integration architecture, see Azure Integration Architecture.

Overview

Azure resources emit two categories of logs:

  • Activity logs — Account-level events such as resource creation, deletion, policy changes, and service health notifications.

  • Resource logs — Service-specific logs emitted by individual resources (for example, Azure SQL query logs, App Service HTTP access logs, or Key Vault audit logs).

Both categories are routed through the same pipeline:

  1. Azure Diagnostic Settings forward logs from each resource to an Azure Event Hub Namespace.

  2. An Azure Function App with an Event Hub trigger reads each batch of events and forwards them to the Kloudfuse logs ingestion endpoint over HTTPS.

  3. Logs appear in the Kloudfuse Log Analytics interface within seconds.

Common use cases:

  • Security monitoring — Detect sign-in anomalies, privilege escalation, and suspicious API calls from Azure Active Directory and resource audit logs.

  • Compliance reporting — Maintain a tamper-evident record of all configuration changes to Azure resources.

  • Incident investigation — Correlate Azure infrastructure changes with application errors using Kloudfuse’s unified log and metrics view.

Prerequisites

  • An Azure subscription with permission to create Event Hubs, Function Apps, and Diagnostic Settings.

  • The Kloudfuse ingestion endpoint URL and an API key. See Ingestion Authentication with API Key for instructions on generating an API key.

  • The Azure CLI (az) installed and authenticated, or access to the Azure Portal.

  • Node.js 18 or later (for the Function App runtime).

Step 1: Create an Event Hub Namespace and Hub

An Event Hub Namespace is the container for one or more Event Hubs. Create a dedicated namespace and hub for Kloudfuse log ingestion.

Create the Namespace

az eventhubs namespace create \
  --name kloudfuse-logs \
  --resource-group <resource-group> \
  --location <region> \
  --sku Standard
bash

Create the Event Hub

az eventhubs eventhub create \
  --name kloudfuse \
  --namespace-name kloudfuse-logs \
  --resource-group <resource-group> \
  --partition-count 4 \
  --message-retention 1
bash

Create a Shared Access Policy

Create a policy that allows both sending (for Diagnostic Settings) and listening (for the Function App):

az eventhubs namespace authorization-rule create \
  --name kloudfuse-policy \
  --namespace-name kloudfuse-logs \
  --resource-group <resource-group> \
  --rights Listen Send
bash

Retrieve the connection string — you will need this when configuring the Function App:

az eventhubs namespace authorization-rule keys list \
  --name kloudfuse-policy \
  --namespace-name kloudfuse-logs \
  --resource-group <resource-group> \
  --query primaryConnectionString \
  --output tsv
bash

Save the output as EVENT_HUB_CONNECTION_STRING.

Step 2: Create an Azure Function App

The Function App reads events from the Event Hub and forwards them to Kloudfuse.

Create the Function App

Create a storage account (required by Function Apps) and the Function App itself. Both names must be globally unique in Azure:

  • Storage account names are 3–24 lowercase alphanumeric characters (for example, kloudfuselogs<suffix>).

  • Function App names become a subdomain of azurewebsites.net (for example, kloudfuse-logs-<suffix>).

az storage account create \
  --name <storage-account-name> \
  --resource-group <resource-group> \
  --location <region> \
  --sku Standard_LRS

az functionapp create \
  --name <function-app-name> \
  --resource-group <resource-group> \
  --storage-account <storage-account-name> \
  --consumption-plan-location <region> \
  --runtime node \
  --runtime-version 18 \
  --functions-version 4 \
  --os-type Linux
bash

Configure Application Settings

Set the Event Hub connection string and the Kloudfuse credentials as application settings:

az functionapp config appsettings set \
  --name <function-app-name> \
  --resource-group <resource-group> \
  --settings \
    "EventHubConnection=<EVENT_HUB_CONNECTION_STRING>" \
    "KF_API_KEY=<your-kloudfuse-api-key>" \
    "KF_URL=<your-kloudfuse-hostname>"
bash

Replace <your-kloudfuse-hostname> with the hostname of your Kloudfuse instance only — no https:// prefix and no trailing slash, for example kloudfuse.example.com. The forwarder constructs the HTTPS connection internally using the hostname.

Deploy the Function Code

Create the function directory structure and deploy the forwarding function:

mkdir -p kloudfuse-azure-fn/kloudfuse-log-forwarder
cd kloudfuse-azure-fn
bash

Create host.json:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingSettings": {
        "isEnabled": true,
        "excludedTypes": "Request"
      }
    }
  }
}
json

Create kloudfuse-log-forwarder/function.json:

{
  "bindings": [
    {
      "type": "eventHubTrigger",
      "name": "eventHubMessages",
      "direction": "in",
      "eventHubName": "kloudfuse",
      "connection": "EventHubConnection",
      "cardinality": "many",
      "dataType": ""
    }
  ]
}
json

Create kloudfuse-log-forwarder/index.js:

const https = require("https");

const KF_API_KEY = process.env.KF_API_KEY || "";
const KF_URL     = process.env.KF_URL     || "";  // hostname only, e.g. kloudfuse.example.com

module.exports = async function (context, eventHubMessages) {
  const messages = Array.isArray(eventHubMessages)
    ? eventHubMessages
    : [eventHubMessages];

  // Normalise each message to a parsed object, then wrap the batch in a JSON array.
  const records = messages.map(m => (typeof m === "string" ? JSON.parse(m) : m));
  const payload = JSON.stringify(records);

  await new Promise((resolve, reject) => {
    const req = https.request(
      {
        hostname: KF_URL,
        port: 443,
        path: "/api/v2/logs",
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "DD-API-KEY": KF_API_KEY,
          "Content-Length": Buffer.byteLength(payload),
        },
      },
      res => {
        let body = "";
        res.on("data", chunk => { body += chunk; });
        res.on("end", () => {
          context.log(`Kloudfuse response: ${res.statusCode}`);
          if (res.statusCode >= 200 && res.statusCode < 300) {
            resolve();
          } else {
            reject(new Error(`Kloudfuse ingester returned ${res.statusCode}: ${body}`));
          }
        });
      }
    );
    req.on("error", reject);
    req.write(payload);
    req.end();
  });
};
javascript

Deploy the function:

cd kloudfuse-azure-fn
func azure functionapp publish <function-app-name>
bash
The Kloudfuse customer scripts repository contains a maintained reference implementation of index.js.

Step 3: Configure Diagnostic Settings

Diagnostic Settings control which Azure resources send logs to the Event Hub.

Retrieve the Authorization Rule Resource ID

Both activity log and resource log diagnostic settings require the full Azure Resource ID of the Event Hub authorization rule. Retrieve it with:

az eventhubs namespace authorization-rule show \
  --name kloudfuse-policy \
  --namespace-name kloudfuse-logs \
  --resource-group <resource-group> \
  --query id \
  --output tsv
bash

Save the output as EVENT_HUB_AUTH_RULE_RESOURCE_ID. It follows the form: /subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.EventHub/namespaces/kloudfuse-logs/authorizationRules/kloudfuse-policy

Activity Logs (Subscription-Level)

Activity logs cover all API operations performed within a subscription. Configure them once per subscription:

az monitor diagnostic-settings subscription create \
  --name kloudfuse-activity-logs \
  --event-hub-name kloudfuse \
  --event-hub-auth-rule "$EVENT_HUB_AUTH_RULE_RESOURCE_ID" \
  --logs '[
    {"category": "Administrative", "enabled": true},
    {"category": "Security",       "enabled": true},
    {"category": "ServiceHealth",  "enabled": true},
    {"category": "Alert",          "enabled": true},
    {"category": "Policy",         "enabled": true}
  ]'
bash

Resource Logs (Per-Resource)

Resource logs must be enabled individually for each Azure resource you want to monitor. The following example enables diagnostic settings for an Azure SQL database:

az monitor diagnostic-settings create \
  --name kloudfuse \
  --resource <resource-id> \
  --event-hub-name kloudfuse \
  --event-hub-rule "$EVENT_HUB_AUTH_RULE_RESOURCE_ID" \
  --logs '[{"categoryGroup": "allLogs", "enabled": true}]' \
  --metrics '[{"category": "AllMetrics", "enabled": false}]'
bash

Replace <resource-id> with the Azure Resource ID of the resource to monitor. Use --logs '[{"categoryGroup": "allLogs", "enabled": true}]' to capture all log categories.

Each Azure service exposes different log categories. See the Azure Monitor resource log categories reference for a full list.

Verify the Integration

Confirm the Function App is Processing Events

  1. In the Azure Portal, open the <function-app-name> Function App.

  2. Click Functions in the left menu and select the kloudfuse-log-forwarder function.

  3. Click Monitor to view recent invocations. Each row corresponds to one batch of Event Hub messages. The Status column must show Success.

  4. Click a recent invocation to view the log output. A line like Kloudfuse response: 200 confirms successful delivery.

Confirm Logs are Arriving in Kloudfuse

  1. In the Kloudfuse UI, click the Logs tab and select Search from the drop-down.

  2. Set the time picker to the last 15 minutes.

  3. In the search bar, click Advanced Search and enter the following FuseQL query to confirm Azure logs are flowing:

    source="azure"
    fuseql

    Any result confirms that Azure log records are reaching Kloudfuse.

  4. To filter by log type, use the category field (set by Azure Diagnostic Settings):

    category="Administrative"
    fuseql

  5. To find all Azure Active Directory sign-in events:

    source="azure" and operationName="Sign-in activity"
    fuseql

  6. To count log volume by Azure resource type:

    source="azure" | count by resourceType
    fuseql

Troubleshooting

Function App Has No Invocations

The Function App is not receiving events from the Event Hub.

  1. Confirm that Diagnostic Settings are configured and pointing to the correct Event Hub Namespace and hub name.

  2. In the Azure Portal, open the Event Hub and click Process data > Explore. Verify that incoming message counts are non-zero after making an API call in your subscription.

  3. Confirm the EventHubConnection application setting in the Function App contains the correct connection string, including the EntityPath=kloudfuse suffix.

  4. Check that the authorization rule has both Listen and Send rights.

Function Invocations Fail with HTTP 5xx

The Function App is delivering to Kloudfuse but receiving error responses.

  1. In the Function App Monitor view, click a failed invocation and check the log for the HTTP status code returned by Kloudfuse.

  2. 401 or 403 — the KF_API_KEY value is incorrect or expired. Regenerate the key in Kloudfuse under Administration > API Keys and update the app setting.

  3. 404 — the KF_URL is incorrect or includes an https:// prefix. The setting must be the hostname only (for example kloudfuse.example.com). Verify the hostname is correct and that the Kloudfuse ingestion endpoint (/api/v2/logs) is reachable from the Function App.

  4. 5xx — the Kloudfuse ingester returned an error. Check the Kloudfuse pod logs: kubectl logs -n kfuse -l app=ingester.

Logs Appear in the Function App but Not in Kloudfuse

  1. Confirm the KF_URL application setting contains the hostname only (for example kloudfuse.example.com) with no https:// prefix and no trailing slash.

  2. Confirm that the Function App can reach the Kloudfuse endpoint. If the Function App runs inside a VNet with restricted egress, add an outbound rule allowing HTTPS to the Kloudfuse IP range.

  3. Check whether the log payload is valid JSON. The function forwards raw Event Hub messages; if Azure emits malformed records, the ingester may reject them silently.

Event Hub Lag is Growing

The Function App is falling behind the Event Hub.

  1. In the Azure Portal, open the Event Hub and view the Consumer Groups lag metric for the Function App consumer group.

  2. If lag is consistently above zero, increase the Event Hub partition count: az eventhubs eventhub update --partition-count 8 …​

  3. Alternatively, increase the Function App plan to a Premium or Dedicated plan to allow more concurrent executions.

External References