Alloy config to get metrics from log files

Grafana Configuration
,
1

  • What Grafana version and what operating system are you using?
    I’m using Grafana cloud with the latest version of Alloy on Ubuntu.

  • What are you trying to achieve?
    I have device health data from many remote devices that check in every few seconds, and this data is aggregated in log files on the Ubuntu server. Each line contains a nested JSON document with values like temperature, CPU usage, disk usage, etc. as well as a timestamp. I’d like to automatically process these log files, extract metrics from each line with the timestamp (and label which device it came from), and store these in Grafana Cloud.

  • How are you trying to achieve it?
    I’m trying to use Alloy to process the logs and forward on to the remote Prometheus write endpoint. I’m new to all of this so I assume I’m doing a few things wrong or don’t understand concepts correctly.

  • What happened?
    I got Alloy to run without an error, but I have yet to see any data show up in Grafana Cloud

  • What did you expect to happen?
    Data to show up either as logs or as metrics.

  • Can you copy/paste the configuration(s) that you are having problems with?
    // Collect logs from a file
    local.file_match “device_logs” {
    path_targets = [{ “path” = “/var/log/myapp/device_health*.log” }]
    }

// Extract specific data from JSON logs, including nested JSON
loki.source.file “device_health” {
targets = local.file_match.device_logs.targets
forward_to = [loki.process.extract_metrics.receiver]
}

// Parsing JSON and sending metrics (CPU, temperature, etc.) from nested JSON
loki.process “extract_metrics” {
stage.json {
// First, parse the outer JSON
expressions = {
health_data = “HealthJSON”,
}
}

// Now parse the nested JSON (HealthJSON)
stage.json {
    source = "health_data"
    expressions = {
        cpu_usage    = "cpu_usage",
        temperature  = "temperature",
        disk_usage   = "disk_usage",
    }
}

forward_to = [loki.write.grafanacloud.receiver]

}

loki.write “grafanacloud” {
endpoint {
url = “https://myapp.grafana.net/loki/api/v1/push”

    basic_auth {
        username = "myusername"
        password = "mypassword"
    }
 }

}

prometheus.exporter.self “example” {}

// Configure a prometheus.scrape component to collect Alloy metrics.
prometheus.scrape “demo” {
targets = prometheus.exporter.self.example.targets
forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

prometheus.remote_write “grafana_cloud” {
endpoint {
url = “https://myapp.grafana.net/api/prom/push”

basic_auth {
  username = "myusername"
  password = "mypassword"
}

}
}

Any help anyone can give with basic Alloy configuration examples or just helping me understand the concepts would be great. It seems like a pretty simple use case, other than the nested JSON, so hopefully it’s an easy fix. My ultimate goal is to set up alerting based on these health values.

2

I am seeing some data now after changing my Loki/Prometheus endpoints to what is specified in the Data sources page on the portal (https://myapp.grafana.net/connections/datasources). Nothing in logs yet, but some metrics are showing up (https://myapp.grafana.net/explore/metrics) - but not anything I’m interested in. These are the kind of metrics I’m seeing: alloy_build_info, alloy_component_controller_evaluating, alloy_component_controller_running_components, loki_source_file_read_bytes_total, prometheus_remote_storage_samples_retried_total.

I was hoping to see cpu_usage, temperature, and disk_usage. Has anyone done something similar - pulling metrics out of JSON in their logs? I assumed this was a pretty common use case but I haven’t seen any example configurations.

3

You are scraping from alloy agent, not the host. To see host metrics use prometheus.exporter.unix (prometheus.exporter.unix | Grafana Alloy documentation)

4

Thanks for your response. I’m not really interested in host metrics, at least not right now. I’m trying to create custom metrics for our devices out in the field like network quality, access point name, and battery percentage. They report their health to our API, and the API logs everything to log files on the host. I assumed I could parse the log files and turn each metric the devices report into time series data. I was able to do something similar with Datadog.

5

I understand I might be doing things in a totally unorthodox way since I’m pretty new to observability. If there’s a better workflow I’m open to other ideas.

7

You need to change signal type: logs to metrics.

Use Grot AI helper | Grafana Labs and ask/follow (it’s AI, so not blindly) “how to make metrics from logs in the alloy”

8

No luck there or with ChatGPT unfortunately. I obviously don’t understand enough to ask the right questions, or the documentation for what I’m trying to do either doesn’t exist or hasn’t been included in the training. If anyone happens to come across this and has gotten something similar working please reply with an alloy config example. To restate - I’d like for there to be a process that automatically goes through my log files, extracts values from the JSON in each line, and sends them to a cloud based time series database where I can create charts and alerts.

9

I don’t personally use this, but you can use loki.process and stage.metrics to turn logs into metrics. See loki.process | Grafana Alloy documentation

Your comment here though:

sounds a lot more dynamic. Like any sort of log parsing, the logs need some structure to them and you’d need to know what you are parsing for from alloy configuration standpoint. If you are looking to parse ad-hoc JSON files and generate metrics dynamically might be easier for you to just write a script to do that.

10

I appreciate your help. Based on my inexperience with these systems and the experience of the past few days, I think I’ll need some serious hand holding to get to resolution here; probably like a phone call and screen sharing. You might be right that it could be easier to write custom scripts, but my data is not going to change much, and I know what the JSON paths will be. My thought was to use an existing system for ease of future scalability.

I’m finally getting log data now (after learning my logs username is different than the metrics username) as well as metrics, but it’s still just the meta metrics for alloy. I’ll leave my current config in case anyone else has a similar question. I feel like this must be somewhat close.

local.file_match "device_logs" {
    path_targets = [{ "__path__" = "/var/log/myapp/device_health*.log" }]
}

// Extract specific data from JSON logs, including nested JSON
loki.source.file "device_health" {
    targets = local.file_match.device_logs.targets
    forward_to = [loki.process.extract_metrics.receiver]
}

// Parsing JSON and sending metrics (CPU, temperature, etc.) from nested JSON
loki.process "extract_metrics" {

    forward_to = [loki.write.grafana_cloud.receiver]

    stage.json {
        expressions = {
            device_id = "DeviceID",
            user_id = "UserID",
            health_json = "HealthJSON",
        }
    }

    stage.json {
        source = "health_json"
        expressions = {
            cpu_temp = "cpu.temp",
            cpu_percent = "cpu.percent",
            gpu_temp = "gpu.temp",
            memory_percent = "memory.virtual_memory.percent",
            disk_percent = "disk.percent",
            wifi_signal = "radio.wi.signal_dBm",
        }
    }

    stage.labels {
        values = {
            cpu_temp       = "cpu_temp",
            cpu_percent    = "cpu_percent[0]",
            gpu_temp       = "gpu_temp",
            memory_percent = "memory_percent",
            disk_percent   = "disk_percent",
            wifi_signal    = "wifi_signal",
        }

    stage.metrics {
      metric.gauge {
        name = "cpu_temperature"
        description = "CPU temperature in Celsius"
        source = "cpu_temp"
        action = "set"
      }
  
      metric.gauge {
        name = "cpu_percent"
        description = "CPU percent core 1"
        source = "cpu_percent"
        action = "set"
      }
  
      metric.gauge {
        name = "gpu_temperature"
        description = "GPU temperature in Celsius"
        source = "gpu_temp"
        action = "set"
      }
  
      metric.gauge {
        name = "memory_usage_percent"
        description = "Memory usage percentage"
        source = "memory_percent"
        action = "set"
      }
  
      metric.gauge {
        name = "disk_usage_percent"
        description = "Disk usage percentage"
        source = "disk_percent"
        action = "set"
      }
  
      metric.gauge {
        name = "wifi_signal_strength"
        description = "WiFi signal strength in dBm"
        source = "wifi_signal"
        action = "set"
      }
    }

    }
}

prometheus.exporter.self "default" { }

prometheus.scrape "alloy_metrics" {
  targets = [{
    __address__ = "localhost:12345",  // This is the default metrics port for Alloy
  }]
  forward_to = [prometheus.remote_write.grafana_cloud.receiver]
}

// Write logs to Loki
loki.write "grafana_cloud" {
    endpoint {
        url = "https://logs-prod-006.grafana.net/loki/api/v1/push"
        basic_auth {
            username = ""
            password = ""
        }
    }
}

// Write metrics to Prometheus
prometheus.remote_write "grafana_cloud" {
  endpoint {
    url = "https://prometheus-prod-13-prod-us-east-0.grafana.net/api/prom/push"

    basic_auth {
      username = ""
      password = ""
    }
  }
}
11

I gave you link on the Grot, not ChatGPT. Grot will give you some ideas. Of course they must be polished, because logs to metrics is not a standard operation.