Loki only displays the first line of multi-line logs, cutting off the remaining lines

Grafana Loki
1

I’ve installed promtail in my K8 cluster using the recommended approach explained here:

Here is my configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: http://v145:3100/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}

      relabel_configs:
        - source_labels: ['__meta_kubernetes_pod_node_name']
          target_label: k8_node
        - source_labels: ['__meta_kubernetes_pod_node_name']
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          source_labels: ['__meta_kubernetes_namespace']
          target_label: k8_namespace
        - action: replace
          source_labels: ['__meta_kubernetes_pod_name']
          target_label: k8_pod
        - action: replace
          source_labels: ['__meta_kubernetes_pod_container_name']
          target_label: k8_container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels: ['__meta_kubernetes_pod_uid', '__meta_kubernetes_pod_container_name']
          target_label: __path__  # DO NOT MODIFY THIS
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels: ['__meta_kubernetes_pod_uid', '__meta_kubernetes_pod_container_name']
          target_label: k8_path

At this point, I am not able to see multi-line logs in Grafana. Only first line is displayed and I thin something in the process cuts the rest of the line.

How can I fix the issue, so I can see multi line logs in Grafana?

2

I don’t see any multiline stage in your config:

1 Like
3

Thank you for help.

The logs are generated by my Python application. I wonder which one of the following is a better or recommended practice:

Solution 1- Modify the Python application to combine multi-line message to one like. So when there is a Python exception, all its stack trace is packed to one line with a delimiter.

Solution 2- Add multiline capability to Promtail’s configuration

4

I would vote for solution 2. Solution 1 is hacking.

1 Like
5

Agree.

Is this a correct syntax or approach?

    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: http://v145:3100/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      pipeline_stages:
        - multiline:
            firstline: ^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]
            max_lines: 128
            max_wait_time: 3s    
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}

      relabel_configs:
        - source_labels: ['__meta_kubernetes_pod_node_name']
          target_label: k8_node
        - source_labels: ['__meta_kubernetes_pod_node_name']
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          source_labels: ['__meta_kubernetes_namespace']
          target_label: k8_namespace
        - action: replace
          source_labels: ['__meta_kubernetes_pod_name']
          target_label: k8_pod
        - action: replace
          source_labels: ['__meta_kubernetes_pod_container_name']
          target_label: k8_container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels: ['__meta_kubernetes_pod_uid', '__meta_kubernetes_pod_container_name']
          target_label: __path__  # DO NOT MODIFY THIS
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels: ['__meta_kubernetes_pod_uid', '__meta_kubernetes_pod_container_name']
          target_label: k8_path
6

It seems that the configuration sample is wrong.

Shouldn’t it use the cri pipeline stage?

Means:

  pipeline_stages:
    - docker: {}

should be

  pipeline_stages:
    - cri: {}