Invalid number of logs

I have nginx logs in JSON format and loki with promtail which imports those logs. Everything worked fine until I notices some strange spikes of traffic in grafana charts. I manually checked log files and found that when actual number of records for specific date is 3 and on Grafana charts I see hundreds of records.

During examination of 'Explore → Loki" section I also found some strange spikes on “Logs Volume” chart as displayed below:

There are no actual log records for this specific date but there are 3524 “logs” found. How can I filter out this records on my charts in Grafana and what are those “logs” actually are?

Here is the sample query for the chart above:

sum by(status) (count_over_time({filename=“/var/log/nginx/access-domain_tld_json.log”, job=“nginx_logs”} | json | error = “” [$__interval]))

seem that problem appeared because there is no mapping to timestamp field in promtail for nginx log format, therefore all this “statistics” build upon number of records imported for some period of time by promtail

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.