JSON Array - Detected Fields

Grafana Loki
1

Hi Everyone,

A Grafana/Loki question. Ingesting JSON data. Does Grafana not support JSON arrays? I believ I saw it in the documentation that it’s not supported?

Grafana only likes this format. It successfully has all the Detected Fields shown in Grafana.

cat file1.json

{ "SourceIP": "192.168.20.1", "DestinationIP": "192.168.20.254", "Username": "JDOE", "Service": "FTP" }
{ "SourceIP": "192.168.20.2", "DestinationIP": "192.168.20.254", "Username": "WDOE", "Service": "HTTP" }
{ "SourceIP": "192.168.20.3", "DestinationIP": "192.168.20.254", "Username": "LDOE", "Service": "FTP" }
{ "SourceIP": "192.168.20.4", "DestinationIP": "192.168.20.254", "Username": "MDOE", "Service": "HTTP" }

This is a proper JSON format. However, Grafan does not like this format. It has no detected fields.

cat file2.json

{ "login": [ { "SourceIP": "192.168.20.1", "DestinationIP": "192.168.20.254", "Username": "JDOE", "Service": "FTP" } ] }
{ "login": [ { "SourceIP": "192.168.20.2", "DestinationIP": "192.168.20.254", "Username": "WJDOE", "Service": "HTTP" } ] }
{ "login": [ { "SourceIP": "192.168.20.3", "DestinationIP": "192.168.20.254", "Username": "LDOE", "Service": "FTP" } ] }
{ "login": [ { "SourceIP": "192.168.20.4", "DestinationIP": "192.168.20.254", "Username": "MDOE", "Service": "HTTP" } ] }
2

Loki does not parse embedded list in json. There are some other logging platforms that can support it somewhat, but you should still limit its use and only do so for certain field instead of entire log line.

Imagine if your log line is such:

{"login": [ {"SourceIP": "123", ...} ] }

Because of the structure of embedded list, the possibility to have multiple entries within one single log line is implied:

{"login": [ {"SourceIP": "123", ...}, {"SourceIP": "456", ...} ] }

At this point you should start to see why it’s not feasible to parse such log structure.

3

Thanks for the response. Understood.

Are there guidelines or mutual understanding in the logging world about not doing embedding lists when it comes to logging?

I’m a rookie regarding JSON. My only thoughts are trying to find the right balance of providing suitable JSON to Grafana and also making the same JSON data usable for other purposes.

Both of my examples can be parsed with JQ, so in theory both formats are usable outside of Grafana.

cat file1.json | jq '. | select(.Service=="FTP")'

{
  "SourceIP": "192.168.20.1",
  "DestinationIP": "192.168.20.254",
  "Username": "JDOE",
  "Service": "FTP"
}
{
  "SourceIP": "192.168.20.3",
  "DestinationIP": "192.168.20.254",
  "Username": "LDOE",
  "Service": "FTP"
}

cat file2.json | jq '.login[ ] | select(.Service=="FTP")'

{
  "SourceIP": "192.168.20.1",
  "DestinationIP": "192.168.20.254",
  "Username": "JDOE",
  "Service": "FTP"
}
{
  "SourceIP": "192.168.20.3",
  "DestinationIP": "192.168.20.254",
  "Username": "LDOE",
  "Service": "FTP"
}
4

I don’t know if there is anything like that, more like operating within the constraints of platforms provided.

Logs are more than just the json body. Your log line might be just a json body, but there are other information. Consider a log line such as:

<TIMESTAMP> <LABEL1>=abc <LABEL2>=def {"login": [ {"SourceIP": "123", ...}, {"SourceIP": "456", ...} ] }

Assuming you are able to parse such log line, what do you do regarding timestamp and the metadata labels?

In this case, it would be simpler and cleaner to change the logline by shifting the login one level down (assuming you care about it) so that each log line is self contained. For example, consider your original logline:

{"login": [ {"SourceIP": "123", ...}, {"SourceIP": "456", ...} ] }

I would parse it on the ingestion side and transform that to:

{"type": "login", "SourceIP": "456", ...}
{"type": "login", "SourceIP": "123", ...}
5

Nice suggestion and good point. Thanks!