Need a new way to label the Alloy remotecfg "ID"

Grafana Alloy
1

I am new to the Grafana world and have been tasked with building out our Grafana Cloud stack and integrating it with our environments. I’ve come across an issue that I can’t seem to come up with a great solution for.

We have 4 environments (Prod, PilotLight, IST, QA) all with the same setup. IST and QA have different server names (ISTAPP01, QAAPP01) but our PRD and Pilotlight all have the same server names (PRDAPP01).

I setup Alloy across our PilotLight environment first and used constants.hostname as the ID to automatically pull the hostname. It worked well until I setup Alloy on 2 PRD servers yesterday and noticed the new Alloy installs were not showing up under my Alloy Fleet Management. I guessed this was due to the ID being the same.

I went into my PilotLight environment and updated the config on those 2 matching servers to change the remotecfg “ID” to “PLPRDAPP01” and I’m now seeing all of my Alloy installs.

Throughout the year we refresh our PilotLight environment by re-standing everything back up from PRD backups. This would undo the static “ID” that we had set in PilotLight.

I’d like to come up do something along the lines of “PL” + constants.hostname for the “ID”. Alloy did not like this and I haven’t been able to find any suggestions online digging through forum posts so thought I’d see if anyone has ran into this issue before.

One issue with my idea above is when we refresh PilotLight from PRD backups, we’re right back to having to update all of the PilotLight Alloy configs to add the PL in front – so not much different than just hardcoding a name in there.

Any other suggestions are welcome! Thanks in advance for anything advice given.

2

I would recommend you to add both hostname and environment labels to all your Alloy agents so that data sent from your Alloy agents can have unique set of labels.

3

Thank you for the reply! How would I go about adding the environment label as well? Posting the remotecfg section below to show you what I have currently:

remotecfg {
url = “https://fleet-management-prod-008.grafana.net”
id = constants.hostname
poll_frequency = “60s”

basic_auth {
	username = "<userid>"
	password = "<apikey>"
}

}

Later on in the config I have this prometheus.relabel section that I set my environment for the server:

prometheus.relabel “integrations_windows_exporter” {
forward_to = [prometheus.remote_write.default.receiver]

rule {
target_label = “environment”
replacement = “PilotLight” // Change this per environment (Production, PilotLight, IST, QA)
}

rule {
source_labels = [“volume”]
regex = “HarddiskVolume.*”
action = “drop”
}

rule {
action = “keep”
regex = “up|windows_cpu_interrupts_total|windows_cpu_time_total|windows_cs_hostname|windows_cs_logical_processors|windows_cs_physical_memory_bytes|windows_disk_drive_status|windows_logical_disk_avg_read_requests_queued|windows_logical_disk_avg_write_requests_queued|windows_logical_disk_free_bytes|windows_logical_disk_idle_seconds_total|windows_logical_disk_read_bytes_total|windows_logical_disk_read_seconds_total|windows_logical_disk_reads_total|windows_logical_disk_size_bytes|windows_logical_disk_write_bytes_total|windows_logical_disk_write_seconds_total|windows_logical_disk_writes_total|windows_net_bytes_received_total|windows_net_bytes_sent_total|windows_net_packets_outbound_discarded_total|windows_net_packets_outbound_errors_total|windows_net_packets_received_discarded_total|windows_net_packets_received_errors_total|windows_net_packets_received_unknown_total|windows_os_info|windows_os_paging_limit_bytes|windows_os_physical_memory_free_bytes|windows_os_timezone|windows_service_status|windows_system_context_switches_total|windows_system_processor_queue_length|windows_system_system_up_time|windows_time_computed_time_offset_seconds|windows_time_ntp_round_trip_delay_seconds|”
source_labels = [“name”]
}
}

That is where I set the environment on each of the servers.

Again, I’m completely new to the Grafana world and these configs, so I very well could be setting the environment in a way that works but possibly not the best way to go about it… if that makes sense?

Thanks so much for your help!

4

You can use external_labels in loki.write to set labels for everything that passes through it, see loki.write | Grafana Alloy documentation.

Our standard practice is to set environment specific labels using this method for each Alloy agent we have. For example (I use a declaration here because we typically re-use the same set of labels for both loki.write and prometheus.remote_write so we don’t have to hardcode it twice):

declare "default_labels" {
  export "external_labels" {
    value = {
      aws_account_alias    = "account123",
      aws_account_id       = "1234567890",
      aws_ec2_instance_id  = "i-instance123",
      aws_region           = "us-east-1",
      environment          = "someprod",
    }
  }
}

default_labels "_this" {}

//
// some loki pipelines
//

loki.write "grafana" {
  external_labels = default_labels._this.external_labels

  endpoint {
    url = "<endpoint>"
  }
}