Remote Agent Promtail - 404 Not Found

[orayt03]

Hello all, we already have configured our loki and remote promtail.

1. Both of the localhost promtail and remote promtail are working and sending updated logs through HTTP. However we need to send encrypted logs over the internet.

2. When using HTTPS on URL in promtail, we are receiving
   level=error ts=2022-02-16T17:59:34.281444845Z caller=client.go:360 component=client host=loki-server.domain:9200 msg="final error sending batch" status=404 error="server returned HTTP status 404 Not Found (404): 404 page not found"

3. Our loki port 3100 is running @ proxy port of 9200.

4. This is the status of curl command from our remote host

HTTP/2 200
server: nginx
date: Wed, 16 Feb 2022 17:50:54 GMT
content-type: application/json
content-length: 58

{"status":"success","data":["__name__","filename","job"]}

5. Configuration of our promtail both localhost and remote

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: 'https://username:key@loki-server.domain:9200/loki/api/v1/push/'

scrape_configs:
  - job_name: remote-nginx
    static_configs:
    - targets:
        - localhost
      labels:
        job: remote-nginx-logs
        __path__: /var/www/*/log/*.log

5. Config file of our loki

server:
  http_listen_port: 3100
  http_listen_address: 127.0.0.1
  http_server_read_timeout: 1000s
  http_server_write_timeout: 1000s
  http_server_idle_timeout: 1000s
  log_level: info

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Thanks!

[b0b]

Hi @orayt03 ,

what proxy do you use and how have you configured that?

[orayt03]

Hi b0b, we solved it by adding the proxy to the data sources of loki.