Multiple Alloy configs via import file and html challenges

Grafana Alloy
, ,
1

Hi All,

I have a couple of challenges (a a newbie), though I got a lot of great thing to work in a good way:

  1. I work against three different environments, each with individual authentication methods, cert and basic.
  2. I have individual Windows and Linux configs.
  3. I have individual app/system/infrastructure configs in a directory structure
  4. While testing configs we use import file, while prod uses import html
  5. Our configs will be contributed to 500+ customers, each with individual OS, systems, infrastructure and apps to monitor.
  6. We use environment variables to set the customer, current environment to which we send the logs, path to configs, hostname etc.

I need to solve at least 2 things:

  1. By use of an env. var, say: ENDPOINT_ENV=Test, I need Alloy to import a config (file) called Test.alloy (the easy part). The challenge is what to put in Test.alloy. I’m aiming at adding all ENDPOINT_ENV relevant import (file|html). Currently we have multiple "declare “fetch” blocks in separate files - one per system/aps/infrastructure to collect information from, and this all works great - except for this:

loki.process “default” {
forward_to = argument.forward_to.value
stage.tenant {
value = string.format(“%s_sys”, coalesce(sys.env(“TENANT_PREFIX”), “cust”))
}
}

Where I have to comment the “stage.tenant” for it to work, as there is no tenant defined in Test, as one of the challenges.
We have one common config file with declare “send”", which among other controls: loki.write “default”. This also works fine except I need two different endpoints - one for certificates and one for basic_auth. Are both configured with different names, one still fails if certs can’t be found.
So what was hoping to do here is using the ENDPOINT_ENV i.e. Test.alloy to help this out, so every core and depending specific detail need for a specific env will be defined in i.e. Test.alloy, and all other imports will use the settings from this config.
So one of the questions are, can we use nested imports (I think the answer is yes)? Doing so (nested), can move to conflicting part from the current declare “send”" config file, so if define ENDPOINT_ENV=Test I can have a declare “send”" part in the Test.alloy for basic auth, vise versa a tls part in the Prod.alloy ?
Or is there a smarter way to control these quite different and dependent environment settings in a way that won’t raise conflicts in the config files, AND still be flexible?

  1. We need to find a way, where Alloy for any given customer ONLY reads(imports) configs that are relevant for the given customer. Here again I thought of using an (maybe additional) environment variable(s), which would hold a list of “things” (configs) to import. How could this be achieved in a good way? PS Octopus Deploy will be used to deploy Alloy to the customers.

Many thanks in advance.

/Bjarne

2

I still hope for some input here :+1:

3

This is a pretty big question, I am not sure if my answer would satisfy you.

  1. With that many clients, I assume you would have some sort of configuration management tool for your client’s computers. If so, in my opinion your use case is sufficiently complex that you will likely get better results if you just handle Alloy configurations from your configuration management tool instead.

  2. If you really want to do this from Alloy, you’ll have to break it down at the block level. You cannot optionally include or exclude a stage within a configuration block (at least I don’t think it’s possible). So using Loki as an example, minimum you’d need a ingress (where the logs come from, this can be file targets or http listener), a process layer (this is probably mostly loki.process), and egress (loki.write), and you’d need all three for each different use case, different OS, and each client.

  3. Of course the most advanced solution would be to do fleet management like Grafana Cloud does. Unfortunately this is not available outside of Grafana Cloud as far as I know, so you’ll have to write your own fleet management server.

1 Like
4

I second the comment of @tonyswumac. I understand that your solution is to use the Fleet Management API. Here’s a GitHub implementation GitHub - opsplane-services/alloy-remote-config-server: Remote server implementation of alloy remote config API. You have to change a variable and adapt it to your needs.