Cannot expand a single line into multiple lines(mvexpand alternative)

[gmsjcb]

We are migrating from splunk to loki-grafana.
While migrating one of our dashboard we hit a road-block.

Problem statement:
Currently, we are generating logs in the following format:

2023-06-09 18:05:42.042 sampleJsonArray=[{“identifier”:“6c928ef1-8cac-4ed3-a855-232ceae17f7d”, “count”: 5}, {“identifier”:“bf9b11ec-721e-47a0-84a4-b3f192c9d6f4”, “count”: 7}, {“identifier”:“c07d9a2e-1219-446e-885f-2b3a302c486c”, “count”: 5}]
2023-06-09 18:05:43.042 sampleJsonArray=[{“identifier”:“6c928ef1-8cac-4ed3-a855-232ceae17f7d”, “count”: 3}]
2023-06-09 18:05:44.042 sampleJsonArray=[{“identifier”:“bf9b11ec-721e-47a0-84a4-b3f192c9d6f4”, “count”: 2}, {“identifier”:“c07d9a2e-1219-446e-885f-2b3a302c486c”, “count”: 8}]

We need to figure out the sum of all the count value per identifier

Identifier                                                      | Count
6c928ef1-8cac-4ed3-a855-232ceae17f7d                            | 8
bf9b11ec-721e-47a0-84a4-b3f192c9d6f4                            | 9
c07d9a2e-1219-446e-885f-2b3a302c486c                            | 13

We did try out some transformation e.g.: ‘Extract Fields’ → JSON
And using table to visualise the result. But this doesn’t work.

I am trying to compute such logs is by expanding each line of sampleJsonArray into multiple lines (each line with same timestamp possibly) with one json object per line (and then use some kind of reduce/group by transform). in splunk we used mvexpand.

I am not able to find any way to expand a row into such multiple rows.
Not sure, if this is even possible with Loki-grafana.

[tonyswumac]

This is, as far as I know, not currently possible with Loki.

I personally have a similar use case, and I had to break the logs on the ingestion side (I used logstash for this).

For example, consider your log:

2023-06-09 18:05:44.042 sampleJsonArray=[{“identifier”:“bf9b11ec-721e-47a0-84a4-b3f192c9d6f4”, “count”: 2}, {“identifier”:“c07d9a2e-1219-446e-885f-2b3a302c486c”, “count”: 8}]

You would have to parse the sampleJsonArray part and turn one log line into multiple log lines depending on number of element:

2023-06-09 18:05:44.042 {“identifier”:“bf9b11ec-721e-47a0-84a4-b3f192c9d6f4”, “count”: 2}
2023-06-09 18:05:44.042 {“identifier”:“c07d9a2e-1219-446e-885f-2b3a302c486c”, “count”: 8}

[timoteus78]

I am not familiar with logstash and a very new with loki-grafana but would it be something to convert your example to something like shown below using if possible with logstash or otherwise something like awk?

2023-06-09 18:05:42.042 description 6c928ef1-8cac-4ed3-a855-232ceae17f7d=5 bf9b11ec-721e-47a0-84a4-b3f192c9d6f4=7 c07d9a2e-1219-446e-885f-2b3a302c486c=5