Promtail can not push log to loki

Grafana Loki
, ,
1

I use promtail to push my docker-compose’s log,but I meet a problem like this:

level=error ts=2023-05-11T01:38:37.421215001Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

It seems like my promtail configured Incorrectly.This is my config:

[root@tq-deploy tq_deploy]# cat loki/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
#    relabel_configs:
#      - source_labels: ['_meta_docker_container_name']
#        regex: '/(.*)'
#        target_label: 'container'
#        replacement: "$1"
#
#      - source_labels: ['__path__']
#        regex: '/var/log/(.*)'
#        target_label: 'job'
#        replacement: "$1"

image
image847×204 9.52 KB

I don’t know how can I push log to loki. My docker-compose use json-file to store log.

2

You have to have something in relabel_configs

3

Hi,this is my config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  #- url: http://loki:3100/loki/api/v1/push
  - url: http://loki:3100/api/prom/push

scrape_configs:
  - job_name: system
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['_meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
     #   replacement: "$1"
    #  - source_labels: ['__meta_docker_container_id']
    #    target_label: '__path__'
    #    replacement: '/var/lib/docker/containers/$1/*.log'
    pipeline_stages:
      - docker: {}
      - cri: {}
      - json:
          expressions:
            stream: stream
            log: log
      - static_labels:
          host: "tq"

and I can find default in promtail default label like ‘__meta_docker_container_id’,but I think promtail not send label to loki.

image
image1424×398 28.7 KB

image
image1021×59 3.87 KB

4

I am a newest of loki and I have no idea about my probelm.

5

Any label that starts with __ is either metadata or has some other special meaning. They will not be added as labels.

Your relabel_config should have a few actions like this

  relabel_configs:
      - action: replace
        source_labels:
          - __meta_something_you_want_as_a_label
        target_label: name_of_label_you_want_to_see_in_loki
1 Like
6

Yes，I am a fool,I write _ as __,it not work. Now everything is ok. thank you!

1 Like
7

I’m having a similar problem myself though I am getting the logs for promtail it is also sending logs like this

2023-11-02 20:42:09.555
trajano_promtail.ic4p6gb3jq91xmqv7cpyep8ij.qsmyx1it80x5lnnjmymouxdb4
stderr
level=error ts=2023-11-03T00:42:09.473271419Z caller=client.go:430 component=client host=loki:3100 msg=“final error sending batch” status=400 tenant= error=“server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream”

Fields
container_name trajano_promtail.ic4p6gb3jq91xmqv7cpyep8ij.qsmyx1it80x5lnnjmymouxdb4
stream stderr

My config file scrape config looks like this

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        filters:
          - name: label
            values: [ "com.docker.swarm.task" ]
    relabel_configs:
      - source_labels: [__meta_docker_container_log_stream]
        target_label: stream
      - source_labels: [__meta_docker_container_name]
        regex: ''
        action: drop
      - source_labels: [__meta_docker_container_name]
        regex: '/(.*)'
        target_label: container_name
      - source_labels: [__meta_docker_container_label_promtail_enabled]
        regex: "false"
        action: drop

So I am getting the replacement but some messages are causing a problem. So I tried to do the drop when __meta_docker_container_name is '' along with a label check

Turning on debug log didn’t help much since it didn’t tell me what message or log entry it sent that causes the issue.

1 Like
8

Hi, I’m having the exactly same issue with a very similar config:

      docker_sd_configs:
        - host: unix:///var/run/docker.sock
          refresh_interval: 5s
      pipeline_stages:
      ...
      relabel_configs:
      - source_labels: ['__meta_docker_container_label_grafana_logging_disable']
        regex: 'true'
        action: drop
     ...

This causes the same error:

level=error ts=2024-01-29T14:31:18.460915591Z caller=manager.go:49 component=distributor path=write insight=true msg="write operation failed" details="error at least one label pair is required per stream"

It does not happen when I remove this bit:

- source_labels: ['__meta_docker_container_label_grafana_logging_disable']
        regex: 'true'
        action: drop

Have you found a solution for this?
Does anybody know whether there any way to drop logs only if a label has a certain value?

9

Okay, turns out the fix was to simply add a static label like so in pipeline_stages instead of adding it in relabel_configs:

- static_labels:
    instance: '<...>'

I don’t quite understand why a dropped log requires a label though.

10

In order to filter containers by their labels, just apply a filter like this:

docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 60s
        filters:
          - name: label
            values: ["promtail.enable=true"]

Static labels work well for static (well, obviously) labels that would be great to add. For example, in my current configuration I add hostname so that in grafana I can filter by this label:

pipeline_stages:
      - static_labels:
            node_hostname: "${HOST_HOSTNAME}"

Full Scrape Configs:

scrape_configs:
  - job_name: containers

    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 60s
        filters:
          - name: label
            values: ["promtail.enable=true"]

    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container_name

    pipeline_stages:
      - static_labels:
            node_hostname: "${HOST_HOSTNAME}"
      - docker: {}
      - timestamp:
          format: RFC3339Nano
          source: time
      - labels:
          stream:
          container_id:
      - output:
          source: log