I have nginx logs in JSON format and loki with promtail which imports those logs. Everything worked fine until I notices some strange spikes of traffic in grafana charts. I manually checked log files and found that when actual number of records for specific date is 3 and on Grafana charts I see hundreds of records.
During examination of 'Explore → Loki" section I also found some strange spikes on “Logs Volume” chart as displayed below:
There are no actual log records for this specific date but there are 3524 “logs” found. How can I filter out this records on my charts in Grafana and what are those “logs” actually are?
Here is the sample query for the chart above:
sum by(status) (count_over_time({filename=“/var/log/nginx/access-domain_tld_json.log”, job=“nginx_logs”} | json | error = “” [$__interval]))