Hello everyone,
My company has been using Grafana for some time, usually just for monitoring license usage for our RLM servers. Recently I’ve been trying to set up a simple alert which when an admin makes a change to our AD it generates an email alert.
The data is scraped by Alloy, and then sent to Loki to be displayed by Grafana.
config.alloy
logging {
level = “debug”// Forward internal logs to the local Loki instance.
write_to = [loki.write.loki.receiver]
}loki.source.windowsevent “security” {
eventlog_name = “Security”
forward_to = [loki.write.loki.receiver]
}loki.write “loki” {
endpoint {
url = format(
“http://%s/loki/api/v1/push”,
coalesce(env(“LOKI_HOST”), “Grafana:3100”),
)
}
}
This worked fine on the first day, but on the second day I realized I wasn’t getting logs anymore. I let it run for another day while I worked on something else and strangely enough another day later my logs from the second day appeared. I set up the alert and my logs continued to come in a day late.
Here’s an image demonstrating this fact, on the left is the time that the log was received by loki, on the right is the time the log was generated in AD, always one day prior.
I have no idea where to go from here, is there a way to check within loki what the transfer status is? Is it unlikely that Alloy and Loki can handle all of the security logs from my DC? I’d appreciate any help that can push me towards near-realtime AD change alerting.
Cheers,
Konrad