Forward logs from files in variable/rotating paths

Grafana Loki
1

Hello! I’m new in Grafana Loki. I have Syslog server which collecting various logs from hardware, network equipment and other systems in my network. All logs destination is:

/var/log/mycompany/{HOST.NAME}/{YEAR}/{MONTH}/{DAY}/{PROGRAM}.log

Do anybody have an idea how to use Promtail and Loki to push collect logs for different dates in this storage scheme? In Zabbix for example I can use RegExp for path to log file and parse all log files, but didn’t find how to perform similar with Loki. It’s not applicable in this case configure promtail for each file, but I really need to navigate log data collected for any period using only Web UI.

In short I’m looking something like this to use in my task, but I have already tried different variations without success.

# Parse all available data
/var/log/mycompany/myhost.domain.local/\d+/\d+/\d+/program.log
# Parse last year
/var/log/mycompany/myhost.domain.local/2024/\d+/\d+/program.log
# Parse last month
/var/log/mycompany/myhost.domain.local/2024/04/\d+/program.log
1 Like
2

You can use ** for recursive look up. In your case it would be /var/log/mycompany/**.log.

3

Yes, I do this on promtail side to discover all log files from all subfolder. But I need filtering option to apply on Grafana side. For example, filter like doesn’t return any data.

{filename="/var/log/mycompany/appserver.mycompany.local/2024/05/**/CRON.log"} |= ``
4

Syntax for regexp label filtering is different. Try&improve (maybe / will need escaping):

{filename=~"/var/log/mycompany/appserver.mycompany.local/2024/05/. */CRON.log"} |= ``