How to implement audit logging?

Grafana Loki
1

I’m trying to replace ELK stack with loki. Elastic has this feature: Enable audit logging | Elasticsearch Guide [8.6] | Elastic

I cannot find anything similar in loki by myself so I tried to ask you guys.

I’m new here so if question (from topic) is stupid then sorry.

For now I think that in loki there cannot be something like that because loki is storing data on some external object stores so it should be this object store which has auditing feature.

The goal is to know who,when accessed particular logs data. I plan to set-up Loki in k8s on DOKS (Digital Ocean Kubernetes Service).

Maybe there are some recommended woarkaround for this?

Thank you all for answers

1 Like
2

You can sort of do audit log, but not directly with Loki.

The first question to answer is how you envision your access pattern to be. Loki itself doesn’t have any UI, so from my personal use case I don’t expect the users to hit the Loki API directly, therefore I instead rely on Grafana + OKTA integration to be the authentication point, and with organizational separation on Grafana based on OKTA group membership, plus auth_enabled enabled on Loki (more explanation below), I can control what users see, and be able to audit their access on Grafana based on which organization they logged in to.

However, if you anticipate users to hit Loki API directly it would probably be quite a bit more involved. Loki has a configuration called auth_enabled. This can be a bit misleading at first, in that it doesn’t actually enable authentication. It just enables the “organizational” feature so that when you send logs to Loki you are required to supply a header called X-Scope-OrgID and logs will now be separated both logically and physically based on the org ID. You can read more about it here: Authentication | Grafana Loki documentation

The best way to approach authentication then is obviously to handle the authentication elsewhere, and map the org ID to user or groups. For example if you were using Nginx it might look like this using local account (there are more examples on github):

http {
  map $remote_user $loki_org_id {
    default unknown;
    user1 org1;
    user2 org2;
    <...>
  }

  server {
    listen               80;
    client_max_body_size 100m;

    location = /loki/api/v1/push {
      auth_basic           "Authenticate";
      auth_basic_user_file /etc/nginx/.htpassword;
      proxy_set_header     X-Scope-OrgID $loki_org_id;

      proxy_pass http://loki$request_uri;
    }

    <Other Loki endpoint>
  }
}

This then gives you the ability to log user activity on Nginx, and you could even go further and integration Nginx with OKTA and map OKTA groups to org IDs (this I haven’t personally tried).

3

as I understand:

  • “audit their access on Grafana” - you mean this one right?
    • if yes then only Grafana Enterprise or Cloud Advanced is possible then right?
  • “based on which organization they logged in to.” - so as I understand if I’ll go with the same solution I won’t be able to say that John made a request to the log but only that organization1 made a request to log1 - am I right here?
    • or I’m wrong because information can be retrieved from OKTA + Grafana like: John accessed Grafana at 03:00 pm (info from OKTA) and Grafana orgId=2 made a request to loki

I think that I fully understand the nginx example - then both authentication & audit logging is on the nginx side and nginx will say to me clearly that John (his IP) was accessing particular endpoint at
timestamp with result …

So with nginx with some additional work I can have at least something similar to Audit events | Elasticsearch Guide [8.11] | Elastic

4

You are sort of correct, and if you are from a heavily regulated environment what we do may not be enough for you. Essentially with our model the audit trail stops at Grafana. You can tell which user access which dashboard, and if they ran an ad-hoc query against Loki, but you can’t necessarily link a user to every query run on Loki directly (and I’d argue you don’t necessarily have to because users can’t hit Loki directly anyway, but this may not fly with some auditors). If you have to link user activity to Loki directly then you’ll probably need to integrate Nginx with your IdP system and use that as your gateway into Loki.

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.