Alloy>Loki Backlogged data source?

Hello everyone,
My company has been using Grafana for some time, usually just for monitoring license usage for our RLM servers. Recently I’ve been trying to set up a simple alert which when an admin makes a change to our AD it generates an email alert.
The data is scraped by Alloy, and then sent to Loki to be displayed by Grafana.

config.alloy

logging {
level = “debug”

// Forward internal logs to the local Loki instance.

write_to = [loki.write.loki.receiver]
}

loki.source.windowsevent “security” {
eventlog_name = “Security”
forward_to = [loki.write.loki.receiver]
}

loki.write “loki” {
endpoint {
url = format(
“http://%s/loki/api/v1/push”,
coalesce(env(“LOKI_HOST”), “Grafana:3100”),
)
}
}

This worked fine on the first day, but on the second day I realized I wasn’t getting logs anymore. I let it run for another day while I worked on something else and strangely enough another day later my logs from the second day appeared. I set up the alert and my logs continued to come in a day late.
Here’s an image demonstrating this fact, on the left is the time that the log was received by loki, on the right is the time the log was generated in AD, always one day prior.

I have no idea where to go from here, is there a way to check within loki what the transfer status is? Is it unlikely that Alloy and Loki can handle all of the security logs from my DC? I’d appreciate any help that can push me towards near-realtime AD change alerting.

Cheers,
Konrad

Disregard I figured it out.
.
.
.
.
.

xpath_query = “Event/System[EventID=4722 or EventID=4722 or EventID=4723 or EventID=4724 or EventID=4725 or EventID=4726 or EventID=4727 or EventID=4728 or EventID=4729 or EventID=4730 or EventID=4738 or EventID=4740 or EventID=4781]”

Too much data, added that to config.alloy and it works now.

1 Like