Why didn't Grafana show all the logs that Loki collected?

Hello everyone, I have some problems here, there is no solution, who can help me.

Collecting Logs(In the case of flannel):

namespace/pod: kube-system/flannel

Promtail configuration:

- job_name: Kubernetes-Flannel
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    regex: 'flannel'
    source_labels:
    - __meta_kubernetes_pod_label_app
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_container_name
    target_label: __service__
  - source_labels:
    - __meta_kubernetes_pod_node_name
    target_label: __host__
  - action: replace
    replacement: $1
    separator: /
    source_labels:
    - __meta_kubernetes_namespace
    - __service__
    target_label: app
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_name
    target_label: pod_name
  - source_labels:
    - __meta_kubernetes_pod_ip
    action: replace
    target_label: pod_ip
  - source_labels:
    - __meta_kubernetes_pod_container_name
    action: replace
    target_label: container_name
  - source_labels:
    - __meta_kubernetes_pod_phase
    action: replace
    target_label: pod_phase
  - replacement: /var/log/pods/*$1/*.log
    separator: /
    source_labels:
    - __meta_kubernetes_pod_uid
    - __meta_kubernetes_pod_container_name
    target_label: __path__
  pipeline_stages:
    - docker: {}

promtail UI:

There are logs in the file.

/var/log/pods/kube-system_kube-flannel-ds-8fblq_185fcd9d-bdd1-42f7-bafa-1ad03392e95d/kube-flannel/0.log

My question:
Why did Grafana not display the Flannel logs?

I think Promtail has collected Flannel’s logs, and I feel the problem is being received by Loki. I changed Loki’s logs level to DEBUG, but no problem can be found. Is there any solution for troubleshooting?

1 Like