Question about time extraction

Grafana Alloy
1

Hi, I am new to Grafana Loki and Alloy, and I am having difficulty extracting the timestamp from the logs.

Here is an example log:

1,2/16/25 13:51,16301012097
1,2/16/25 13:50,16301012112

I have written this regex and tested it in a regex tester, but it does not work in Alloy. It is still showing the log injection time. Can I get some advice? Thanks.

  • Alloy, version v1.6.1
  • Grafana v11.5.2
loki.process "add_labels_timestamp" {
  stage.static_labels {
    values = {
      job  = "job1",
      service_name = "service1",
    }
  }

  stage.regex {
    expression ="^(?P<sn>[^ ]+)\\,(?P<timestamp>\\d+\\/\\d+\\/\\d+\\s\\d+\\:\\d+)\\,(?:.*)"
  }

  stage.timestamp {
    source = "timestamp"
    format = "2/16/25 13:50"
  }

  forward_to = [loki.write.grafana_loki.receiver]
}
2

When using a custom format you need to reference a specific time (see time package - time - Go Packages).

Try changing your format to 1/02/06 15:04

3

Thanks. After changing the config to 1/02/06 15:04, the new log cannot be found in any selected time range. I have tried selecting the entire years 2001, 2002, 2006, 2016, and 2025, but no new log can be found.

Any suggestions for troubleshooting? Thanks a lot.

4

Anything from Loki logs , any errors

5

Thanks for the hint.

I checked the logs using the command journalctl -f -u loki | grep sample.log and saw an error mentioning ‘has timestamp too old.’ This proves that the regex should be working.

Then, I added reject_old_samples and reject_old_samples_max_age to the Loki config file.

limits_config:
  metric_aggregation_enabled: true
  reject_old_samples: false
  reject_old_samples_max_age: 1y

After restarting Loki and adding a sample log to the sample.log file, I no longer see the ‘timestamp too old’ error, but I still cannot search for the log in Grafana.

6

Hi all, after further study, I understand that the label is affected by the log timestamp. When a label is mapped to a log with a newer timestamp, that label no longer accepts logs with older timestamps, even if the reject_old_samples setting is enabled.

I attempted to create a new set of Alloy configuration with a new log filename and label, then appending old logs one by one in ascending order of timestamp to the new log file. No ‘timestamp too old’ error appeared because the labels were new. The new filename and label appeared in Grafana Explore quickly, but the logs still could not be found.

After a period of time, with numerous service restarts and trial and error (such as appending new logs with no timestamp), the logs suddenly became searchable in Explore.

old_timestamp

I cannot understand the behavior, and I don’t even know how to reproduce it. Let’s see if someone knows what is happening or can help troubleshoot it together. Thanks a lot.

7

Look at this thread it might help . Issue was on Loki side

8

Thanks, everyone. I have tried the whole process again, and it seems that I have found the reason.

Overall, the settings are correct. The issue is caused by server performance and the sequence of timestamps in the logs.

It is better to first sort the logs by timestamp in ascending order, and then find a machine with better performance for testing.

1 Like