Security grafana-loki

[freerabbit]

I have a question about securing grafana-loki. The promtailclients deliver their logs over port 3100 to grafana-loki, the same port i use to connect grafana. How can I make sure that not everyone who has access to port 3100 of grafana-loki can also mount the data source in grafana?

[tonyswumac]

Couple of security measures we took for our clusters:

• Loki cluster is fronted by Nginx reverse proxy with local users. The users are consider API users, each dedicated to a Loki tenant. There are a lot of things you can do here depending on how strict you want to be. You can create pair of API users per tenant, read can be used by Grafana, write can be used by promtail. But we did not go with that. we use one API user for both read and write.

• No user access to Loki cluster directly. We built a frontend job if users need to export large quantity of logs.

• Grafana has to go through Nginx proxy as well. This allows you to create one data source per tenant, and be able to limit access to each tenant based on how your Grafana permission is controlled.

I know there are more extreme measures you can take if you want to be more strict.

[freerabbit]

Thank you very much, but how can I prevent that every host that provides data via promtail can also use the grafana-loki as a data source for grafana?

[tonyswumac]

I can think of two ways to address that:

1. This is what we do for our environments. Use a log aggregator of sorts and have the authentication happen there. You would have to maintain that access to the aggregator is granted only to designated support personnel. And the flow looks like this:

host (promtail, no auth) → log aggregator (auth) → loki

In terms of what to use for the aggregator, you can actually use another promtail there (pretty sure you can, we actually didn’t test), or, because Loki is HTTP based, you could simply setup a forward proxy with just an nginx instance. Here is what our configuration looks like for the aggregator (as you can see, it simply forwards to Loki with a simple auth):

http {
  sendfile     on;
  tcp_nopush   on;

  server {
    listen 80;

    resolver <LOCAL_DNS_SERVER>;
    set      $loki_upstream "https://loki-endpoint.local:443";

    location / {
      proxy_set_header Host "log-aggregator-nginx";

      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_pass       $loki_upstream;
      proxy_set_header Authorization "Basic <USERNAME:PASSWORD_AND_ENCODED>"; # This needs to be base64 encoded
    }
  }
}

This has the added benefit that you don’t have to worry about secure distribution of password to all hosts, instead you just need to worry about one set of servers (the log aggregators).

2. You can also be more granular on your user management on the nginx frontend for Loki. Generate read users or write users, and block write traffic for read users, and read traffic for the write users.