Ingest old log file into loki using Alloy, timestamp issue

Grafana Alloy
1

I am trying to ingest some old logs for analysis trying to preserve the the log line date and time. I have tried everything i can think of without success. I am able to get the date and time in loki but it does use it for the entry date and time; i.e will show all the log entries in current date and time when I explore the logs or use Drilldown.

Is there any method to make loki use the log line date time , below is what I have but it does not work.

// 3. Process the logs, Extract data from log messages and add labels
loki.process “syslog_process” {
forward_to = [loki.write.default.receiver]

// Syslog formats can vary widely, so you may need to adjust this regex

stage.regex {
expression = ^(?P<timestamp>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})\s+(?P<hostname>\S+)\s+(?P<logfile>\S+)\s+thread:(?P<thread>\S+)\s+priority:(?P<priority>\w+)\s+category:(?P<category>\S+)\s+exception:\s*(?P<exception>.+)$
labels_from_groups = false
}

// Set the timestamp based on the extracted field

stage.timestamp {
source = “timestamp”
format = “DateTime”

}

stage.labels {
values = {
timestamp = “”,
hostname = “”,
priority = “”,
thread = “”,
category = “”,
exception = “”,
}
}

}

in loki I have added the following

limits_config:
metric_aggregation_enabled: true
enable_multi_variant_queries: true
reject_old_samples: false
reject_old_samples_max_age: 1y
unordered_writes: true

Is this even possible?

Any help is much appreciated..

1 Like
2

From kapa-ai in grafana ask ai slack channel

You relax (or disable) the “reject old samples” check **in Loki’s limits_config**, not in Alloy. ### Option 1 – Allow older logs by increasing the max age Increase reject_old_samples_max_age:

yaml limits_config: reject_old_samples: true reject_old_samples_max_age: 336h # 14 days, for example

This raises the cutoff so Loki accepts older log timestamps before returning greater_than_max_sample_age.[[greater_than_max_sample_age](Enforce rate limits and push request validation | Grafana Loki documentation); [Troubleshoot ingest](Troubleshoot log ingestion (WRITE) | Grafana Loki documentation)] ###

Option 2 – Disable the check (not recommended)

yaml limits_config: reject_old_samples: false

This turns off the age check entirely, so Loki won’t reject logs for being too old.[[Troubleshoot ingest](Troubleshoot log ingestion (WRITE) | Grafana Loki documentation); [limits_config ref](Grafana Loki configuration parameters | Grafana Loki documentation)] ### Per‑tenant override You can also change these per tenant via the **runtime configuration file** (overrides section), instead of globally.[[greater_than_max_sample_age](Enforce rate limits and push request validation | Grafana Loki documentation); [limits_config ref](Grafana Loki configuration parameters | Grafana Loki documentation)]

3

Thanks, but I need the loki to recognize the original date time, currently loki uses it’s own date time on that is the ingest date time, I need the actual date time from the log lines.

4

No matter what I do with stage.timestamp it still using the tike loki received the line i.e Time is the current date instead of the date and time from the log line. Is this even possible?

5

image
image563Ă—234 22.1 KB

6

I don’t think DateTime is a valid format reference. See loki.process | Grafana Alloy documentation