I have free tier of Grafana Cloud. I wanted to send logs to Loki by Promtail. This is my docker compose for Promtail:
version: "3"
services:
promtail:
image: grafana/promtail:2.5.0
volumes:
- /etc/promtail/config.yml:/etc/promtail/config.yml
- /var/log/app:/app/logs
command: -config.file=/etc/promtail/config.yml -config.expand-env=true
And this is my Promtail config file:
server:
http_listen_port: 0
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
client:
url: https://<user>:<password>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: app
__path__: /app/logs/*.log
If I run Promtail on server I am receiving errors:
2022-07-29T22:58:41.046124553Z level=warn ts=2022-07-29T22:58:41.045587747Z caller=client.go:349 component=client host=logs-prod-eu-west-0.grafana.net msg="error sending batch, will retry" status=-1 error="Post \"https://<user>:***@logs-prod-eu-west-0.grafana.net/loki/api/v1/push\": context deadline exceeded"
When I run it from localhost, it works (from container and by curl). If I run API with curl from server, it works. If I run Promtail with dry run mode from docker container, it works:
cat /app/logs/app.log | promtail --stdin --dry-run --client.url https://<user>:<password>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push
Clients configured:
----------------------
url: https://:<user>:<password>@logs-prod-eu-west-0.grafana.net/loki/api/v1/push
batchwait: 1s
batchsize: 1048576
follow_redirects: false
backoff_config:
min_period: 500ms
max_period: 5m0s
max_retries: 10
timeout: 10s
tenant_id: ""
stream_lag_labels: filename
level=info ts=2022-07-29T23:01:05.966685041Z caller=server.go:260 http=[::]:80 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2022-07-29T23:01:05.967562785Z caller=main.go:119 msg="Starting Promtail" version="(version=2.5.0, branch=HEAD, revision=2d9d0ee23)"
2022-07-29T23:01:05.968599967+0000 {hostname="a92fcdf485cd", job="stdin"} 2022-07-29 23:39:27,927 [INFO] No new posts founded.
2022-07-29T23:01:05.968606686+0000 {hostname="a92fcdf485cd", job="stdin"} 2022-07-29 23:39:27,928 [INFO] Script took: 0.18s.
I tried to set batchwait to 10s, it did not help. When I POST logs with curl, it works from localhost. I cannot test curl from inside the container because I cannot install curl.
It looks like problem is with docker but cannot determine why, on localhost, Promtail in docker works fine.