I want to add labels to the logs, before sending them to loki, from promtail, based on the log text. Following is my promtail_config code
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://Loki_IP/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
pipeline_stages:
- regex:
expression: '.*Test.*'
labels:
contains_test: "test"
But, it is not able to add Labels to the logs containing the world Test. Any idea whats wrong with it? I have also tried following
pipeline_stages:
- match:
selector: '{job="varlogs"}'
stages:
- regex:
expression: '.*Test.*'
- labels:
contains_test: "true"
Both of the approaches are not working
You need to use the match block to perform a conditional label add. See match | Grafana Loki documentation
Yes, I have tried that also. You can see my 2nd solution, but still its not working for me? I have tried this approach
pipeline_stages:
- match:
selector: '{job="varlogs"}'
stages:
- regex:
expression: '.*Test.*'
- labels:
contains_test: "true"
That’s also incorrect. What you are doing is matching job = "varlogs". Instead you should do regex first, with a group capture, then match the group to a string, then set label there.
A line or two of example logs would also be helpful.
Do you have any working example of it? If yes, can you please share the code snippet?
If you can give me a couple of samples I can come up with something.
Sample means? Like console logs? I am doing POC, so I create system logs like with logger, i.e. logger "Testing123"
Here is my sample log:
this is a Test log
this is a normal log
this is a test log but lower case
Promtail configuration:
---
server:
disable: true
positions:
filename: /tmp/positions.yml
clients:
- url: http://localhost:3100/
scrape_configs:
- job_name: test
static_configs:
- targets:
- localhost
labels:
__path__: /tmp/test.log
job: test
pipeline_stages:
- match:
selector: '{job="test"} |= "Test"'
stages:
- static_labels:
contains_test: "true"
Dry run results:
# promtail -config.file=/tmp/promtail.conf --dry-run --inspect
Clients configured:
----------------------
url: http://localhost:3100/
batchwait: 1s
batchsize: 1048576
follow_redirects: true
enable_http2: true
backoff_config:
min_period: 500ms
max_period: 5m0s
max_retries: 10
timeout: 10s
tenant_id: ""
drop_rate_limited_batches: false
stream_lag_labels: ""
level=info ts=2023-10-23T18:59:15.319808277Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=92d67002d7c42a4b6647ee2841d00198
level=info ts=2023-10-23T18:59:15.32223744Z caller=main.go:174 msg="Starting Promtail" version="(version=2.9.2, branch=HEAD, revision=a17308db6)"
level=warn ts=2023-10-23T18:59:15.322358666Z caller=promtail.go:260 msg="disable watchConfig" reason="promtailServer cast fail"
[inspect: static_labels stage]:
{stages.Entry}.Entry.Labels:
-: {filename="/tmp/test.log", job="test"}
+: {contains_test="true", filename="/tmp/test.log", job="test"}
2023-10-23T18:59:20.323526786+0000 {contains_test="true", filename="/tmp/test.log", job="test"} this is a Test log
2023-10-23T18:59:20.324894507+0000 {filename="/tmp/test.log", job="test"} this is a normal log
2023-10-23T18:59:20.324897111+0000 {filename="/tmp/test.log", job="test"} this is a test log but lower case
level=info ts=2023-10-23T18:59:20.323145568Z caller=filetargetmanager.go:361 msg="Adding target" key="/tmp/test.log:{job=\"test\"}"
level=info ts=2023-10-23T18:59:20.323269625Z caller=filetarget.go:313 msg="watching new directory" directory=/tmp
ts=2023-10-23T18:59:20.323493051Z caller=log.go:168 level=info msg="Seeked /tmp/test.log - &{Offset:0 Whence:0}"
level=info ts=2023-10-23T18:59:20.323530403Z caller=tailer.go:145 component=tailer msg="tail routine: started" path=/tmp/test.log
This worked for me. Thanks @tonyswumac
