Ad hoc filters variable for Loki structured_metadata

Grafana
, ,
1

“Ad hoc filters” variable works great for Loki labels and fields, but not with structured metadata. I can transform queries only in the following way (these do not work because company_id is in structured metadata):

  1. {app="main-app"} → {app="main-app", company_id="123"}
  2. {app="main-app"} | json → {app="main-app"} | json | company_id="123"

But I can’t achieve the following:
{app="main-app"} → {app="main-app"} | company_id="123"

Why can’t I use the second example? Because I work with GBs, if not TBs, of data, and using | json becomes very expensive. Ideally, I’d like to avoid unnecessary parsing, especially before filtering.

Can I somehow work around this? I simply need an optional set of filters for fields for structured metadata.

2

That indicates that company_id is not medata, but it is a field in the JSON formatted logs. Make sure that your log pipeline ingest company_id as a structured metadata, when you want to use it in that way.

3

Thank you for the response. I am pretty sure company_id is part of structured_metadata. I can query it if I do {app="main-app"} | company_id="123". What I am trying to do is the same with Ad hoc filters, but there it can only filter as a label or field

4

OK, so if this is working, then it is metadata. Ad hoc filters work with labels only, not with metadata.

5

I understand that this is my exact problem. I am trying to build optional filters for structured metadata. Doesn’t have to use Ad hoc filters, can be variable, or whatever. Basically if I have query like this {app="main-app"} and then I specify company_id:123, I want to transform my query to {app="main-app"} | company_id="123" but if I don’t specify it, query should remain as {app="main-app"}. I was curious if there is some workaround or hack on how I can do that.

6

You can use the dashboard variable (e.g. text type, name filter), where the user can write that filter (part of it, e.g. only company id) and use it in the panel query e.g.

{app="main-app"} | company_id="${filter:raw}"
7

Ok, but then if the filter is not provided, the query turns into
{app="main-app"} | company_id="", which is not going to match anything. I could specify .+, but then, I am filtering with no filter, degrading the query performance. The idea is that the filter only exists when the variable has a defined value.

8

Yeah, be clever and specify the variable for your case, e.g. there must be a full filter value, not just id, and then you can use panel query:

{app="main-app"} ${filter:raw}

And yes, there are billions and billions ™ “buts”:

  • but the user can write invalid filter
  • but user doesn’t know ids
  • …

Why don’t you just use the label, when it makes better sense for your use case?

9

Unfortunately, I can’t use labels since they have quite high cardinality, and indexing them will make Loki unqueriable. {app="main-app"} ${filter:raw} seems the most promising thing I can do, the problem is this will not have a good UX for the average dashboard user, who might not even know what Loki is. For me, though, that would be a perfect go-to solution.

What I am trying to achieve here precisely is the following:

  1. Have a textbox variable for company_id
  2. Somehow transform it into a filter if the value is provided, and if not, into an empty string
  3. Use the result variable in the query

My question is if there is any convenient way to do this; if not, the worst case scenario, I can write a variable plugin that handles this logic, but ideally, if there is some way to play around with variables and achieve this with already existing Grafana functionality, that would be the best.

10

Try/test/improve:

image
image605×1084 45 KB

11

I figured out this very unpleasant workaround that fits my use case. I used a random MySQL datasource to template a filter variable based on the values of other variables (company_id, user_id, and session_id).
Here is how the template logic looks:

SELECT
  CASE
    WHEN '${company_id}' <> '' THEN
      CONCAT(
        '| company_id="${company_id}"',
        CASE WHEN '${user_id}' <> '' THEN ' and user_id="${user_id}"' ELSE '' END,
        CASE WHEN '${session_id}' <> '' THEN ' and session_id="${session_id}"' ELSE '' END
      )

    WHEN '${user_id}' <> '' THEN
      CONCAT(
        '| user_id="${user_id}"',
        CASE WHEN '${session_id}' <> '' THEN ' and session_id="${session_id}"' ELSE '' END
      )

    WHEN '${session_id}' <> '' THEN
      '| session_id="${session_id}"'

    ELSE
      ''
  END

And here is the result:

image
image1952×172 14.6 KB

This will work for now, but I hope to find a better solution at some point.

1 Like