Hello!! I would first like to apologize for using the wrong terminology in my questions. I am new to Grafana Alloy and Loki! I need to monitor syslogs accross my network. I am currently using loki.source.syslog to collect syslogs and write them to loki. How can I apply custom labels based on the ip_address sending the syslog?
I have used discovery.http to send targets and custom labels to prometheus.scrape, but how do I do that with loki? There are no targets and everything references maps.
I know that I can use loki.relabel to change or update labels, but can I use that to add custom labels based on the ip_address in the syslog? It seems like this is to manipulate existing labels in the log.
I am not trying to add dozens of labels, just an application name, location, and environment.
Although it’s not documented, from the source code it looks like a label is added with the remote IP (see alloy/internal/component/loki/source/syslog/internal/syslogtarget/transport.go at b957de68821926812b261df81ac80fcbeadddb05 · grafana/alloy · GitHub)
So you should be able to add a label like this:
loki.process "{NAME}" {
forward_to = ["{...}"]
stage.static_labels {
values = {
remote_ip = "__syslog_connection_ip_address",
}
}
}
Thank you Tony!!!
To confirm, stage.static_labels will add a label called remote_ip to the log and that can be used to search in loki, correct?
Is there a way to take that value and add labels for environment, application, and location? For example, I have the following json,
[
{
“remote_ip”: “1.2.3.4.5”,
“location”: “chicago”,
“environment”: “development”,
“application”: “grafana”
}
]
I want to add the other labels to the log based on the remote_ip.
I looked at loki.relabel and it looks like you can drop, keep, and modify, but not add new labels. Would I use stage.label?
Thank you again for your help!!
Apologies, my previous comment was actually incorrect. You want to use stage.labels to assign value to a label from another label, and use stage.static_labels to hardcode value to a label. So you’d want:
loki.process "{NAME}" {
forward_to = ["{...}"]
stage.labels {
values = {
"remote_ip" = "__syslog_connection_ip_address", # this assign value of __syslog_connection_ip_address label to remote_ip
}
}
stage.static_labels {
values = {
"location" = "chicago",
"environment" = "development",
"application" = "something",
}
}
}
Could I use something like this?
loki.process “add_labels” {
forward_to = [“{…}”]
stage.labels {
values = {
“remote_ip” = “__syslog_connection_ip_address”,
}
stage.match {
selector = “{remote_ip="1.2.3.4.5"}”
stage.static_labels {
values = {
“location” = “chicago”,
“environment” = “development”,
“application” = “app1”,
}
}
stage.match {
selector = “{remote_ip="9.2.3.4.5"}”
stage.static_labels {
values = {
“location” = “chicago”,
“environment” = “production”,
“application” = “app1”,
}
}
stage.match {
selector = “{remote_ip="4.5.6.7.8"}”
stage.static_labels {
values = {
“location” = “indianapolis”,
“environment” = “development”,
“application” = “app2”,
}
}
}
I also have same use case can you help me.
So i have Linux and Windows Server, I want to add Below Static Values/Labels
- Location
- IP Address
- Rack
How can i achieve them and added them to loki and prometheus?
On Fleet Overview Screen I want to Display Location and IP Address
Are you using syslog as well? syslog has a list of labels that it tries to produce internally, which should cover ip address, but for other external information such as rack you’d need to supply those yourself.
I noob in this part.
Please guide me how can i use syslog.
For Rack and Location in Agent I had added global labels
Please guide me the same how can I add Lables for that instance. They are static value. I can manually input it it’s okay.
and what about syslog ?
How to use that ?
can you help me how can i enable syslog ?
prometheus.exporter.self "integrations_alloy" { }
discovery.relabel "integrations_alloy" {
targets = prometheus.exporter.self.integrations_alloy.targets
rule {
target_label = "instance"
replacement = constants.hostname
}
rule {
target_label = "alloy_hostname"
replacement = constants.hostname
}
rule {
target_label = "job"
replacement = "integrations/alloy-check"
}
}
prometheus.scrape "integrations_alloy" {
targets = discovery.relabel.integrations_alloy.output
forward_to = [prometheus.relabel.integrations_alloy.receiver]
scrape_interval = "60s"
}
prometheus.relabel "integrations_alloy" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "(prometheus_target_sync_length_seconds_sum|prometheus_target_scrapes_.*|prometheus_target_interval.*|prometheus_sd_discovered_targets|alloy_build.*|prometheus_remote_write_wal_samples_appended_total|process_start_time_seconds)"
action = "keep"
}
}
prometheus.remote_write "metrics_service" {
endpoint {
url = "{METRICS_URL}"
}
}
loki.write "grafana_loki" {
endpoint {
url = "{LOGS_URL}"
}
}
discovery.relabel "integrations_node_exporter" {
targets = prometheus.exporter.unix.integrations_node_exporter.targets
rule {
target_label = "instance"
replacement = constants.hostname
}
rule {
target_label = "job"
replacement = "integrations/node_exporter"
}
}
prometheus.exporter.unix "integrations_node_exporter" {
disable_collectors = ["ipvs", "btrfs", "infiniband", "xfs", "zfs"]
filesystem {
fs_types_exclude = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
mount_points_exclude = "^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+)($|/)"
mount_timeout = "5s"
}
netclass {
ignored_devices = "^(veth.*|cali.*|[a-f0-9]{15})$"
}
netdev {
device_exclude = "^(veth.*|cali.*|[a-f0-9]{15})$"
}
}
prometheus.scrape "integrations_node_exporter" {
targets = discovery.relabel.integrations_node_exporter.output
forward_to = [prometheus.relabel.integrations_node_exporter.receiver]
}
prometheus.relabel "integrations_node_exporter" {
forward_to = [prometheus.remote_write.metrics_service.receiver]
rule {
source_labels = ["__name__"]
regex = "node_scrape_collector_.+"
action = "drop"
}
}
loki.source.journal "logs_integrations_integrations_node_exporter_journal_scrape" {
max_age = "24h0m0s"
relabel_rules = discovery.relabel.logs_integrations_integrations_node_exporter_journal_scrape.rules
forward_to = [loki.write.grafana_loki.receiver]
}
local.file_match "logs_integrations_integrations_node_exporter_direct_scrape" {
path_targets = [{
__address__ = "localhost",
__path__ = "/var/log/{syslog,messages,*.log}",
instance = constants.hostname,
job = "integrations/node_exporter",
}]
}
discovery.relabel "logs_integrations_integrations_node_exporter_journal_scrape" {
targets = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
rule {
source_labels = ["__journal__boot_id"]
target_label = "boot_id"
}
rule {
source_labels = ["__journal__transport"]
target_label = "transport"
}
rule {
source_labels = ["__journal_priority_keyword"]
target_label = "level"
}
}
loki.source.file "logs_integrations_integrations_node_exporter_direct_scrape" {
targets = local.file_match.logs_integrations_integrations_node_exporter_direct_scrape.targets
forward_to = [loki.write.grafana_loki.receiver]
}
loki.source.syslog "LABEL" {
listener {
address = "LISTEN_ADDRESS"
}
...
forward_to = RECEIVER_LIST
}
I have seen this but i am not able to integrate can you please help
