Unsupported character in tenant id

I’ve been using tenant id template CLUSTER_ID:LOG_TYPE in loki 2.9.
I’m testing to upgrade my loki to 3.0, and I had an error from loki writer that msg="error getting tenant id" err="tenant ID '...' contains unsupported character ':'"

It looks like tenant id validation rule in loki 3.0 is different from loki 2.9.
I looked around loki 3.0 release note, and upgrade guide, but I couldn’t find anything related to this issue. There are too many code changes to 3.0, so I can’t find it in the source code too.

Is this an intended change? If it is, Is there any workaround to use : character in tenant ID?

The validation error is likely from this: dskit/tenant/tenant.go at main · grafana/dskit · GitHub

I looked at the history of the dskit and loki commits, and looks (to me) like the validation function hasn’t changed since forever, so I am not sure how you got it to work in Loki 2.9. Hopefully someone more knowledgable than I can comment.

Below is my opinion. Do you really need LOG_TYPE as part of your tenant ID? It would seem like that would make a good label (in fact I’d even argue that cluster_id should be a label as well and reserve tenant for organizational separation, but I digress). If you aren’t 100% against making the change, perhaps this is an opportunity to do so?

1 Like

I wrote that just LOG_TYPE, but there are different kinds of logs and metrics, like CLUSTER:mimir CLUSTER:audit-logs CLUSTER:tempo, CLUSTER:syslog and so on. I believe it’s not a good idea to store them in a single tenant because they have very different labels and log formats.

I’m using s3 as a loki storage. Can I rename loki tenant by renaming s3 directory?

I don’t think so, but you can test it out on a dev cluster if you have one.

Also, just to address this comment:

There is nothing inherently wrong to store logs with different labels and formats in a single tenant. We have tenant that has json logs, text only logs, java logs, multi-line logs, you name it. But of course it’s your decision and there are more than one ways to do things. Just my opinion.

1 Like

Of course we can store all container logs, syslogs, audit logs and any kind of logs in single tenant, but we are serving multi-TB logs(in compressed size), and it costs us too much resources in indexing and querying.
We had to minimize cardinality of log labels and volumes by subjects.
As we can’t control all of our query clients, grafana should provide more loki query options like mandatory log labels in query to store and query all of them in single tenant. We can’t afford to let clients to query all kinds of logs in single query right now.
Also, some of our logs are business critical, and some logs must be retained by gov regulations, so If loki doesn’t provide a way to rename or migrate tenant id, we might be stuck in 2.9 for long time.

You might consider posting your question in the community slack as well, and see if someone else there has some ideas.

How long do you have to keep logs for? You might consider perhaps operating another Loki cluster with modified tenant ID, and gradually migrate logs over by changing your logging agent to send logs to new cluster with updated ID… Once logs are migrated over, you can then scale down the old cluster to reduce cost, until your regulatory requirements are met then you can get rid of it altogether.