Store Alloy Configs on Central Server

Grafana Alloy
1

Looking to centrally manage all alloy configuration files for my environment in a git repo for version control. I have windows and linux endpoints, some with docker containers running as well.

import.git works okay, but will cause alloy to crash if the matching file is not found.

The best I can come up with is to deploy alloy to my clients with a generic alloy.config like so:

logging {
  level = info
}

import.http "main" {
  url = "https://myserver.lan/" + contants.hostname + ".alloy"
}

main.config "default" { }

then the <hostname>.alloy wouldn’t exist yet, so my HTTP server would serve an “empty” config as a response, allowing it to import without error:

declare "config" { }

and when I update the repo to configure for that host (ex. a linux host):

declare "config" {
  prometheus.exporter.unix "integrations_node_exporter" {
    enable_collectors = ["meminfo"]
  }

  prometheus.scrape "integrations_node_exporter" {
    scrape_interval = "15s"
    targets    = prometheus.exporter.unix.integrations_node_exporter.targets
    forward_to = [prometheus.remote_write.local.receiver]
  }

  prometheus.remote_write "local" {
    endpoint {
      url = "http://prometheus:9090/api/v1/write"
    }
  }
}

which would then get pulled by the client and imported and ran if the syntax is all good, and if there is an error then it wouldn’t update, but also wouldn’t crash alloy.

Ideally I’d like to make all of this modular, since a lot of hosts would share the same components. Like if all linux hosts would import, say, a linux.alloy config as well, which would always have the prometheus.exporter.unix stuff. Then any custom configurations would be included in the <hostname>.alloy config like monitor custom log files or whatever.

How would one go about this? Do I just make one massive config.alloy with tons of declare blocks and then call the ones I need? Or do I make placeholders like so:

import.http "main" {
  url = "https://myserver.lan/" + contants.hostname + ".alloy"
}

import.http "second" {
  url = "https://myserver.lan/" + contants.hostname + "_2.alloy"
}

import.http "third" {
  url = "https://myserver.lan/" + contants.hostname + "_3.alloy"
}

import.http "linux" {
  url = "https://myserver.lan/linux.alloy"
}


main.config "stuff" { }
second.config "stuff" { }
third.config "stuff" { }
linux.config "stuff" { }

Maybe I am overlooking a feature that does this, or just overthinking the problem. Any guidance or ideas are welcome. Cheers!

2

Alloy does support importing configuration, but it’s not very straight forward and can be a little confusing. When it works it’s great, but I’d recommend you to consider the effort and benefit before splitting your config too much.

Here is an example. Let’s say you have a default baseline configuration that includes the default configuration for your prometheus cluster, and you have linux.conf with configuration specifically for linux servers and you want to plug that into your baseline configuration (mock code, not tested).

Let’s say your baseline configuration (config.alloy) looks like this:

prometheus.remote_write "mimir_forwarder" {
  endpoint {
    url = "<STRING>"
        
    basic_auth {
      username = "<STRING>"
      password = "<STRING>"
    }
  }
}

And you want to import a linux metrics configuration that uses that resource prometheus.remote_write.mimir_forwarder as destination. You would then write your linux.conf like this:

declare "linux" {
  argument "metrics_output" {
    optional = false
    comment  = "Where to send collected metrics."
  }

  prometheus.exporter.unix "default" {
    procfs_path    = "/host/proc/"
    rootfs_path    = "/host/root/"
    sysfs_path     = "/host/sys/"
    udev_data_path = "/host/run/udev/data/"
  }

  prometheus.scrape "default" {
    scrape_interval = "10s"
    targets         = prometheus.exporter.unix.default.targets
    forward_to      = [argument.metrics_output.value]
  }
}

Then import it in base configuration config.alloy like so:

remote.http "linux" {
  <CONFIG>
}

import.string "linux_import" {
  content = remote.http.linux.content
}

linux_import.linux "this" {
  metrics_output = [prometheus.remote_write.mimir_forwarder.receiver]
}
1 Like