I am trying to ship logs from local promtail instance to a remote loki and grafana setup. I keep getting the following error:
level=warn ts=2020-12-16T12:58:32.454345353Z caller=client.go:241 component=client host=some-remote-loki-ip-over-vpn:3100 msg="error sending batch, will retry" status=-1 error="Post http://some-remote-loki-ip-over-vpn:3100/loki/api/v1/push: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
- If I try shipping same log using postman or insomnia, then I am successfully able to push them to remote loki instance
- I thought maybe promtail container is unable to access the remote endpoint but curl and ping work as expected. So promtail is able to find the remote loki instance
- I tried searching online for a similar post, found a few (not loki specific) where people suggested updating DNS to 126.96.36.199 but that didn’t help
server: http_listen_port: 9080 grpc_listen_port: 0 positions: filename: /env/app/output/promtail/positions.yaml clients: - url: http://some-remote-loki-ip-over-vpn:3100/loki/api/v1/push scrape_configs: - job_name: system static_configs: - targets: - localhost labels: __path__: /env/app/input/**/*.log