Loki.process will not rewrite timestamp

Grafana Alloy
1

Hello community. I have been at this for days. I cannot get loki to take the timestamp from my log files. It will only use scrape time. I have read dozens of threads and examples on this topic and can’t seem to find the magic. I have output my regex extracted timestamp as a label to troubleshoot that it is extracting ok which it is, so I know the loki.process is working. Below is my snippet as well as a log file line example. Any help is greatly appreciated.

log line:
2025-05-28 05:56:23.616 -07:00 [INF] == Lender import File processing started
my timetsamp label outputs:
2025-05-28 05:56:23.616

loki.process “labels” {

stage.regex {
expression = (?P<timestamp>(?:\d{4}|\d{2})-\d{2}-\d{2} \d{2}:\d{2}:\d{2}(\.|,)\d{3})(.*$)
}

stage.timestamp {
source = “timestamp”
format = “RFC3339”
fallback_formats = [“2006-01-02 03:04:05.000”,“2006-01-02 03:04:05,000”]
}

// Testing only
stage.labels {
values = {
timestamp = “”,
}
}

stage.static_labels {
values = {
service_name = “Bryanmgmt01”,
environment = “Bryanmgmt01”,
}
}

stage.output {
source = “message”
}

forward_to = [loki.write.loki.receiver]

}

2

Finally got this working with correct timestamp. Found one issue in my config. I needed to change 12 hour time to 24 hour time. so now I have the 15 instead of the 03 for the hour:
fallback_formats = [“2006-01-02 15:04:05.000”,“2006-01-02 15:04:05,000”]

Also, I was using older logs with older timestamps for testing and for some reason loki doesn’t like this even if I set “tail_from_end” to false. A bit concerning I can’t pull in older log lines. Maybe there is a better way for it to set that point it thinks it left off at? What if I want to ingest an entire logfile even if it is days old?

3

There are two conditions for sending old logs into Loki:

  1. Make sure reject_old_samples_max_age is configured according to your needs.
  2. Loki will not accept older logs in a log stream if there are new logs already present (log stream is defined as logs with the same set of labels). So if you set Alloy to not tail from end, Loki will accept all the logs up to reject_old_samples_max_age assuming you don’t already have logs in the same log stream.
4

So if I had older timestamps (several hours old) in a completely new logfile name it did take them in and parse timestamp correct, but if they were 2 days old it did not. Is there a limit to how far back loki will ingest logs based on timestamp?

5

ahhh, ok, thank you @tonyswumac for that max age setting. I will play with that.

7

one more update for anyone else following along. I fixed the timezone my browser was showing by adding the location in my alloy config for the stage.timestamp. using the actual timezone that the logs are created in.

stage.timestamp {
source = “timestamp”
format = “RFC3339”
fallback_formats = [“2006-01-02 15:04:05.000”,“2006-01-02 15:04:05,000”]
location = “America/Los_Angeles”
}