I am having different services running in different pods with promtail as sidecar.
Each service generates log files, which the sidecar promtail is aware about using static_configs. But when it comes to the grafana, labels like pod name, namespace, container name are missing. what I would like to see is, each log line extracted from the log file also have the namespace, pod name and container name information in labels.
Config that I tried
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki.loki.svc.cluster.local:3100/loki/api/v1/push
scrape_configs:
- job_name: audit
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: replace
replacement: $1
source_labels: ['__meta_kubernetes_namespace']
target_label: namespace
static_configs:
- labels:
logType: audit
__path__: /var/log/audit/*log
app: app1
The above config doesnt show any namespace label. What should the config look like to get namespace label properly in grafana