Hi,
I have been playing with loki for a few days now. I realized that live logs (ie. system logs ingested regularly) get patterns automatically created.
I ingested some historical logs using promtail and on those logs I dont see any pattern generated (see picture). Based on the logs, visually, I am pretty sure patterns could have been created.
Is there anything I can do to have patterns generated?
Are you sure? Default Loki config doesn’t allow to ingest historic logs usually, e. g. one day old. So check your Loki configuration and Loki/Promtail logs for more details. I guess there will be some errors that logs are dropped due to too old timestamp or something similar.
You can ingest historic logs if you set reject_old_samples: false in limits_config. I can see the data, log volume etc but no patterns is generated.
limits_config:
retention_period: 4320h
deletion_mode: filter-and-delete
metric_aggregation_enabled: true
max_query_length: 0h # Default: 721h
ingestion_rate_mb: 1024
ingestion_burst_size_mb: 5024
reject_old_samples: false
max_line_size: 0
You cannot inject older logs to a log stream that already has newer logs, regardless of what setting you use. Log stream is considered as logs with the same set of labels.
Thanks for your info, I am not trying to inject older logs into a log stream with newer logs. I had logs that were old, let say from 2024, promtail took those logs and loki ingested them without any issue. My problem is not in loading logs into loki at this point but why patterns are not generated.
Here patterns are not generated but each lines are the same, i believe a pattern should have been generated no?
On this recent syslog log, patterns are generated :
How this pattern thingy working? I think this pattern generation is pretty cool and can be very useful so why they are generated for some logs and not others?
There is no pattern being generated. The graph from explorer is simply a count over time on number of logs from your query.
From your first screenshot I do see counts being generated, not sure why you said it’s not. Check your log’s actual timestamp against the graphs, not whatever that’s in your log string.
In the picture showing syslog above, you can see that “patterns” are generated, it is not a simple log volume like in the other pictures. I put in a red square on the picture to show you what I am talking about, one has “Patterns at 0” and one has Patterns 17".
I need to understand how this pattern generation works because it is clearly not working as I am expecting.
I see what you mean. I don’t use that feature myself, so I can’t really answer your question. Hopefully others can chime in.
Quick read on the doc implies that the patterns is from your queries, but I am not entirely sure.
Thanks for your time.
Really not sure that comes from my query. The query is simple “service_name=XXX” for all above cases.
For a log like syslog, loki will find many fields but for others would find only 1 field or even 0. I guess loki has some kind of predefined words and try to detect those in logs lines, then based on that would generate patterns. If the log does not contains those words then no pattern is generated?
If someone in this community could explain how this patterns generation works that would be really appreciated.
For the fields part i think i understand, loki is looking at something like <fieldname>=<somevalue>, if there is nothing like that, no field will be detected.
Remains now to understand the patterns…
And I think I understood how patterns work as well.
Patterns are also looking at <fieldname>=<value> and so the following is generated for promtail log in syslog:
<_> localhost promtail[734]: level=debug ts=<_> caller=filetargetmanager.go:312 msg="new target" labels="{__address__=\"<_>\"}"
the pattern is using field like :
ts=
msg=
labels=
Another example with mysqld log, a field “server_uuid=” is detected in the line:
2024-12-26T08:25:06.620372Z 100 [Warning] [MY-011809] [Server] Cannot replicate to server with server_uuid='99e0b1ce-4b31-11ec-b10d-005056ba5a75' because the present server has purged required binary logs. The connecting server needs to replicate the missing transactions from elsewhere, or be replaced by a new server created from a more recent backup. To prevent this error in the future, consider increasing the binary log expiration period on the present server. The missing transactions are '05a23ca3-4b30-11ec-85d0-005056ba2359:1-6, c9ee6ce1-4b2f-11ec-89ea-cb439e82165e:217841376-220600080'.
Since this happens only once, I guess that’s why no pattern was generated. So, the fields and patterns are both using logfmt key-value formatting for the detection and generation of patterns.
I was hoping the patterns generation would have helped to identify similar log lines and possibly detect a recurring issue. There is maybe another way to identify similar lines in logs, if you know please advice.
Thanks a lot.
Finally, for people having the same issue with free-form logs, the idea is to reformat the log into key-value logfmt style.
The way to do that in promtail is:
- match:
selector: <a selector>
stages:
- regex:
#Get the parts you want from the logs
expression: '^(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\S+)\s\s(?P<level>\S+)\s[0-9]+\s[-]+\s\[(?P<component>.+)-\d+\]\s(?P<class>[^ ].+)\s+\:\s(?P<msg>.+)'
- timestamp:
source: timestamp
format: "2006-01-02 15:04:05.000"
location: "Asia/Tokyo"
#Level should be lower case
- template:
source: level
template: '{{ ToLower .Value }}'
#Replace the line to make it logfmt style
- replace:
expression: '(.*)'
replace: 'time="{{ .timestamp }}" level="{{ .level }}" msg="{{ .msg }}" host="{{ .server }}" component="{{ .component }}" class="{{ .class }}"'
Delete the data in loki
Restart promtail and reload historical data (or just get new data)
Patterns are getting generated.
Hope this helps someone.