Unable to fetch logs from remote server to going my loki

Grafana Loki
1

Hi all, I’m new to Grafana and found out that there’s a log aggregation using Loki with the help of Promtail. I was able to install and fetched logs from my localhost. I would want to get logs from remote servers (i.e: nginx logs from a remote nginx web server) and push it to my grafana server. What could be the possible way of doing it? Thanks so much!

From my server:
my-promtail-config.yaml
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /data/loki/positions.yaml
clients:

Remote server (not sure if i have to add promtail and push to my grafana server):
server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /data/loki/positions.yaml
clients:

2

@sanlupam03 hi, thanks for your question.

You’ll need to install promtail on every host that you intend to fetch logs from. These promtail instances should all be configured to push to the same central Loki installation.

3

Thanks, I have already promtail installed to my remote server

Remote server’s promtail config:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /data/loki/positions.yaml

clients:

  - url: http://loki-server-domain-name:3100/loki/api/v1/push
scrape_configs:
  - job_name: nginx
  static_configs:
    - targets:
        - localhost
  labels:
    job: nginx
    __path__ : /var/log/nginx/*.log
    host: remote-logs

Remote server promtail service message:
promtail-linux-amd64[29850]: level=error ts=2022-02-11T08:45:59.23847224Z caller=client.go:360 component=client host=loki-server-domain-name:3100 msg=“final error sending batch”

Thanks

4

That’s not enough for me to go on, I’m afraid.
Are you sure your promtail server can communicate with your Loki server on that port?

5

Hi, on localhost, promtail server can talk to loki server

Loki server config:

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

promtail localhost config:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /data/loki/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: nginx
  static_configs:
  - targets:
      - remote-server-ip
    labels:
      job: nginx
      __path__: /var/log/nginx/*.log
      host: grafana
6

But the problem is with your remote promtail, right?

7

Hi, ah, yes the problem was on the side of remote promtail / remote server connecting to our local loki.

Output of curl from our remote host/server.

*   Trying loki.server.ip.address...
* TCP_NODELAY set
* Connected to loki-server.domain (loki.server.ip.address) port 3100 (#0)
> GET /loki/api/v1/push HTTP/1.1
> Host: loki-server.domain:3100
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 405 Method Not Allowed
< Date: Fri, 11 Feb 2022 12:07:50 GMT
< Content-Length: 0
<
* Connection #0 to host loki-server.domain left intact
8

Hi, I think I found the error of my remote promtail but currently observing it.

I didn’t notice that config on client was set to:

client:
  url: http://loki-server.domain:3100/loki/api/v1/push

then I changed to:

clients:
  - url: http://loki-server.domain:3100/loki/api/v1/push
9

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.