Loki logs splitting or truncating after 16k~17k characters. [promtail agent running in k8s]

here is my promtail config, please suggest the solution.

server:
  log_level: info
  http_listen_port: 3101

clients:
  - url: http://XXX/loki/api/v1/push
    tenant_id: XXX

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  # See also https://github.com/grafana/loki/blob/master/production/ksonnet/promtail/scrape_config.libsonnet for reference
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - source_labels: [__meta_kubernetes_namespace]           
        action: keep           
        regex: staging|dev|qa|default
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
        - namespace
        - app
        target_label: job
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_uid
        - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
        - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
        - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
        - __meta_kubernetes_pod_container_name
        target_label: __path__

What’s the limits_config portion of Loki configuration looking like?

@tonyswumac there are no limits applied on log length at loki’s end, when I ingested same log from separate linux machine (not from k8s/docker), logs were as similar as at the source, there was no splitting or truncate. I think its an issue with the docker.

1 Like

@tonyswumac its an open issue i guess. can you suggest me any alternative.

Sounds like it’s a Docker problem. I personally have not had to tackle this, so I am not sure how much help I can be. But I’d recommend trying a couple of things:

  1. The GitHub issue listed some multi-line configurations, worth a try.
  2. Try using json-file log driver (which writes to host) and see if the split logs are joined there.
  3. Try fluentd docker log driver. I vaguely recall some mention of fluentd being able to join split logs. Thought I could be wrong.