Promtail getting 400 Bad Request from Loki with no error description

Grafana Loki
,
1

Hi, I am deploying a Promtail log collector on AWS ECS on EC2, who will send collected logs to a Loki server. Right now, I can get the Promtail running and I can access the Promtail server as well as the Loki server. However, the Promtail logs are showing the error below

level=error ts=2024-07-02T20:02:53.566999735Z caller=client.go:430 component=client host:443=loki.example.com msg=“final error sending batch” status=400 tenant= error=“server returned HTTP status 400 Bad Request (400): ”

The error message does not show a specific error reason like those I have seen online, but instead have a tag. I am using awslogs to temporarily inspect the Promtail logs for now. What could be the reason that the error reason is hidden in the logline?

Below is my configuration file, I can confirm on Promtail server that the path is correctly mounted into the container. I don’t see how the configuration could cause 400 Bad Request as well. Also attached is a screenshot of Service Discovery on Promtail server. Thanks!

server:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: debug

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki.example.com:443/loki/api/v1/push

scrape_configs:
  - job_name: host
    static_configs:
      - targets:
          - localhost
        labels:
          job: promtail
          __path__: /mnt/docker_logs/**/*.log

Screenshot 2024-07-02 at 4.37.09 PM
Screenshot 2024-07-02 at 4.37.09 PM2282×720 50.3 KB

2

Is your url http or https? Usually port 443 is reserved for https.

Regardless, you should take a look at your Loki server logs and see why the request is considered bad.

3

I have the same issue when sending logs from Promtail to Loki over HTTPS. Maybe you can disable HTTPS and try sending the logs again from Promtail using HTTP. I was able to send logs to Loki after changing to HTTP.

4

Thanks for the answers! I changed from HTTP to HTTPS, I no longer receive the error. I wish the error message were a little bit clearer. But for anyone who are getting error message, it’s likely the http vs https setting got wrong.