I’ve been running a Loki instance for some time now with around 20 nodes pushing logs into it. Now I am wondering, how I can protect the logs of other clients against an attacker who compromises one of my 20 clients?
If I understand correctly, currently the data sent by any node is fully trusted, so after compromising client A, an attacker could push additional logs into loki using the hostname of node B. If sensible data is injected, this could obscure the real logs of node B up to the point that I cannot discern them any more.
Is there a way to tell loki that it should add client information to log entries? If loki would for example add a field for the IP address that sent the log line, it would be trivial to detect spoofed logs. Even better would be to tell loki that IP w.x.y.z can only send logs as hostname “xyz”. Is anything like this possible?
In my opinion we should not expect Loki to add any sort of metadata, because Loki simply can’t know for sure. There are so many things that can obscure metadata it’s just not reliable. For example, if you are load balancers, proxy, or a log aggregation cluster, any of those could distort that information.
If you are concerned about security, I would recommend you to enable authentication on your Loki cluster, if you haven’t, and use an Nginx frontend with authentication. Then you can create one user per host (probably a bit over kill, we do one user per AWS account). This will prevent any one compromised host to affect others, and you can easily disable / remove the account for that one compromised host / environment.
I already have a separate basic auth password set up for each client, and while this easily allows me to revoke access, this does not prevent a compromised machine from faking logs for other machines.
Loki needs to authenticate the data or add a field that the client cannot control, like the received ip address (which can be simply be passed through by nginx or any load balancers). It’s just important that there is at least one piece of information that a client has no control over.
I disagree that this isn’t reliable. Imagine Loki would always store the X-Real-IP header for any log line sent to it. Whatever reverse proxy or load balancer you use, you can trivially ensure that it is set to the IP address of the true client. And it is something that the true client can never control. Of course you move some trust to the the load balancer or reverse proxy in that case, but this is infinitely better than what we have now. Currently any client is 100% trusted and this would remove trust from the countless clients and put the trust to just a few critical points.
Mutual TLS can be used - you will have own CA and you will issue dedicated client cert for each client. Then, when you receive logs you will compare details from used client cert with hostname. You will pass that log to Loki only if used cert belongs to reported hostname. Otherwise you will drop it.
I wouldn’t expect that can be implemented in Loki directly, but nothing stopping you to have some mutual TLS proxy in front of Loki, which will be doing log verification/authorization.
I fear this also misses the point - it’s not about an external attacker that tries to interact maliciously with your loki instance. This is about the scenario when one of your actual clients get pwned and the attacker now has full control over that client (and thus also the certificate).
My question is how I can protect against such a rogue instance from spoofing the logs of other instances since Loki doesn’t authenticate what it receives. I want to contain any damage to the client that is compromised and prevent an attacker from affecting logs of other instances - which with my current understanding seems impossible.
Not a problem. Host A is using client cert A and you will have this association stored in your business logic. So hack the host A and start to send logs with client cert A on behalf of host B. Authorization logic will detect unapproved combination client cert A/logs of host B and drop those logs.
It is mutual TLS (not plain TLS - https, which you have on the browser usually) - client knows identity of the server, but also server knows identity of the client.
Modern asymmetric cryptography and security is there.
It is mutual TLS (not plain TLS - https, which you have on the browser usually) - client knows identity of the server, but also server knows identity of the client.
I am aware, the important part is just how I can verify this information using loki. To me it doesn’t matter if this is done via TLS certificates or by verifying the IP address in e.g. a wireguard mesh (IP addresses are authenticated). Both of these are authenticated via some kind of preshared secret or identity.
Authorization logic will detect unapproved combination client cert A/logs of host B and drop those logs.
What authorization logic are you referring to? Which options exactly are responsible for this in loki? This is the logic that I wasn’t able to find. If this really exists, then it would indeed solve my problem.
I said that I believe that this is not implemented in the Loki. But nothing stopping you to build own authorization service in front of Loki and Loki will have only role of dummy storage.
Technically nothing is stopping anybody from implementing something like that. But apparently nobody needs this? But given that loki is such a popular and big product I somehow (maybe naively) assumed it would have a solution for this already.
It’s basically the same situation as with push vs pull backups. From my experience, organizations are usually are very concerned about their backups and want to ensure that no attacker comes along and overrides or tampers with their backups of other machines, so they use pull backups or push backups with privilege separation (client can only add backups for itself). Isn’t logging just as critical?
Use the right tool for right job. I have case where dev are printing live tokens into logs. Should I be demanding from Loki devs to implement something to remove token from the logs (or implement some fragile regexp exclusion)?
I would say better is to have right tool, e.g. DLP in this case, which will clean all sensitive information from logs, which then I can store in dumb log storage.
Loki is log storage in the first place and you need authentication/authorization. Do you think that it easy to implement right? Nope, so I gave you simple recommendation: sort it outside of Loki. That “implementation” can be just Nginx with a few lines of Lua in front of Loki.
Anyway, you are asking for enterprise feature (I would say), so I would recommend to have enterprise Grafana/support.
I don’t think anyone should be demanding anything from the Loki devs (and to be clear I’m not expecting them to do anything). And I hear you, so I’m probably going to have to do something like that for my deployment. Thanks a lot for your input.
Ideologically I strongly believe that security should never be a gated feature, and definitely not something just for enterprises. Everybody needs security, enterprises have other needs for which they can and will pay that doesn’t make user instances inferior in their security. So in case Grafana ever considers implementing a security feature like this one I hope they think similarly.
Is there any Loki competitor, which has this problem solved and available for free?
