Loki api can't get labels

[yifanzhuo135]

I deploy promtail as daemonset in my local k8s cluster,and it has already ran successfully.But I can’t get labels when I use loki API to get labels.

[tonyswumac]

Can you share your configuration, please?

Also take a look at the kubernetes_sd_config if you aren’t already using it: Configure Promtail | Grafana Loki documentation

[yifanzhuo135]

apiVersion: v1
kind: ConfigMap
metadata:
name: promtail-cm
namespace: logging
labels:
app: promtail
data:
region: <REGION_NAME>
stage: test
apiKey: <API_KEY> # Don’t remove complement character(“=”)
instanceUrl: <LOKI_INSTANCE_URL>
lokiInstanceID: <LOKI_INSTACE_ID> # Should be int string example for “1234566”
promtail-config: |
server:
http_listen_port: 0
grpc_listen_port: 0

clients:
  - url: https://${INSTANCE_ID}:${API_KEY}@${INSTANCE_URL}/loki/api/v1/push

scrape_configs:
  - job_name: vkube-pod-logs
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_log_host]
        separator: ;
        regex: default;loki
        action: keep

      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
        action: replace
        target_label: __path__
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/var/log/pods/$${1}_$${2}_$${3}/$${4}/*.log'

      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      # app name should be equal to deployment name
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container
      - source_labels: [__meta_kubernetes_pod_uid]
        target_label: pod_uid
      # Add node name as a label
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node_name
       # Add host IP as a label
      - source_labels: [__meta_kubernetes_pod_host_ip]
        target_label: host_ip
      - source_labels: []
        target_label: region
        replacement: ${CLUSTER_REGION:-unknown}
        action: replace
      - source_labels: []
        target_label: stage
        replacement: ${STAGE:-test}
        action: replace

      # Drop the 'filename' label
      - action: labeldrop
        regex: 'filename'
  - job_name: vcloud-pod-logs
    kubernetes_sd_configs:
      - role: pod
    pipeline_stages:
      - docker: {}
    relabel_configs:
      # Keep all pods in the production namespace
      - source_labels: [__meta_kubernetes_namespace]
        regex: vcloud
        action: keep

      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_name, __meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
        action: replace
        target_label: __path__
        regex: '(.+);(.+);(.+);(.+)'
        replacement: '/var/log/pods/$${1}_$${2}_$${3}/$${4}/*.log'

      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
      # app name should be equal to deployment name
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_container_name]
        target_label: container
      - source_labels: [__meta_kubernetes_pod_uid]
        target_label: pod_uid
      # Add node name as a label
      - source_labels: [__meta_kubernetes_pod_node_name]
        target_label: node_name
       # Add host IP as a label
      - source_labels: [__meta_kubernetes_pod_host_ip]
        target_label: host_ip
      - source_labels: []
        target_label: region
        replacement: ${CLUSTER_REGION:-unknown}
        action: replace
      - source_labels: []
        target_label: stage
        replacement: ${STAGE:-test}
        action: replace
      # Drop the 'filename' label
      - action: labeldrop
        regex: 'filename'

Thank you.This is the config of promtail.

[tonyswumac]

Don’t see anything obviously wrong, but I would recommend you to try a couple of things:

1. What do your logs look like? Are logs actually being sent to Loki?

2. Get rid of all relabel configurations and keep perhaps one, so it’s easier to troubleshoot.

[yifanzhuo135]

@yifanzhuo135

- source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_log_host]
        separator: ;
        regex: default;loki
        action: keep

I don’t know whether the logs are sent to loki or not.I can’t see the label in grafana.

[tonyswumac]

1. You can use static_labels to add arbitrary labels into your log streams, would be a good way to test (see static_labels | Grafana Loki documentation).

2. Try to run promtail from cli in debug / inspect mode. See Troubleshooting Promtail | Grafana Loki documentation.

3. It does look like your logs aren’t being sent at all, so I’d double check and make sure promtail has access to your docker / kubernetes logs.