Promtail can not push log to loki

Grafana Loki
, ,
1

I use promtail to push my docker-compose’s log,but I meet a problem like this:

level=error ts=2023-05-11T01:38:37.421215001Z caller=client.go:430 component=client host=loki:3100 msg="final error sending batch" status=400 tenant= error="server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream"

It seems like my promtail configured Incorrectly.This is my config:

[root@tq-deploy tq_deploy]# cat loki/config.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
#    relabel_configs:
#      - source_labels: ['_meta_docker_container_name']
#        regex: '/(.*)'
#        target_label: 'container'
#        replacement: "$1"
#
#      - source_labels: ['__path__']
#        regex: '/var/log/(.*)'
#        target_label: 'job'
#        replacement: "$1"

image
image847×204 9.52 KB

I don’t know how can I push log to loki. My docker-compose use json-file to store log.

2

You have to have something in relabel_configs

3

Hi,this is my config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  #- url: http://loki:3100/loki/api/v1/push
  - url: http://loki:3100/api/prom/push

scrape_configs:
  - job_name: system
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['_meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'
     #   replacement: "$1"
    #  - source_labels: ['__meta_docker_container_id']
    #    target_label: '__path__'
    #    replacement: '/var/lib/docker/containers/$1/*.log'
    pipeline_stages:
      - docker: {}
      - cri: {}
      - json:
          expressions:
            stream: stream
            log: log
      - static_labels:
          host: "tq"

and I can find default in promtail default label like ‘__meta_docker_container_id’,but I think promtail not send label to loki.

image
image1424×398 28.7 KB

image
image1021×59 3.87 KB

4

I am a newest of loki and I have no idea about my probelm.

5

Any label that starts with __ is either metadata or has some other special meaning. They will not be added as labels.

Your relabel_config should have a few actions like this

  relabel_configs:
      - action: replace
        source_labels:
          - __meta_something_you_want_as_a_label
        target_label: name_of_label_you_want_to_see_in_loki
1 Like
6

Yes，I am a fool,I write _ as __,it not work. Now everything is ok. thank you!

1 Like
7

I’m having a similar problem myself though I am getting the logs for promtail it is also sending logs like this

2023-11-02 20:42:09.555
trajano_promtail.ic4p6gb3jq91xmqv7cpyep8ij.qsmyx1it80x5lnnjmymouxdb4
stderr
level=error ts=2023-11-03T00:42:09.473271419Z caller=client.go:430 component=client host=loki:3100 msg=“final error sending batch” status=400 tenant= error=“server returned HTTP status 400 Bad Request (400): error at least one label pair is required per stream”

Fields
container_name trajano_promtail.ic4p6gb3jq91xmqv7cpyep8ij.qsmyx1it80x5lnnjmymouxdb4
stream stderr

My config file scrape config looks like this

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        filters:
          - name: label
            values: [ "com.docker.swarm.task" ]
    relabel_configs:
      - source_labels: [__meta_docker_container_log_stream]
        target_label: stream
      - source_labels: [__meta_docker_container_name]
        regex: ''
        action: drop
      - source_labels: [__meta_docker_container_name]
        regex: '/(.*)'
        target_label: container_name
      - source_labels: [__meta_docker_container_label_promtail_enabled]
        regex: "false"
        action: drop

So I am getting the replacement but some messages are causing a problem. So I tried to do the drop when __meta_docker_container_name is '' along with a label check

Turning on debug log didn’t help much since it didn’t tell me what message or log entry it sent that causes the issue.