Parse json to make a new field

Grafana Loki
1

Hi All,

Probably this is not related to Loki application, but in my environment I use grafana cloud agent to feed file which contains json strings like this:

{"identity": "alpha-vm01", "qname": "triatn.org.ua.", "rrtype": "AAAA", "query-type": "-", "source-ip": "159.203.16.10", "message": "CLIENT_RESPONSE", "family": "INET", "protocol": "UDP", "source-port": 17497, "length": 122, "timestamp": "2020-12-13T16:12:58.367258+00:00", "type": "response", "rcode": "NOERROR", "id": 38286, "country": "US", "city": "Clifton"}

My grafana cloud agent config looks like:

  - job_name: dnstap_receiver
    static_configs:
    - targets:
      - localhost
      labels:
        job: 'dnstap_receiver'
        __path__: '/var/log/dnstap.log'
    pipeline_stages:
    - match:
        selector: '{ job="dnstap_receiver" }'
        stages:
        - json:
            expressions:
              timestamp: timestamp
              identity: identity
              rrtype: rrtype
              message: message
              rcode: rcode
              country: country
        - labels:
            message:
            identity:
            rrtype:
            country:
            rcode:
        - timestamp:
            source: timestamp
            format: RFC3339

Is it possible to make a new json key with value which can be include first and second layer of domain?

I would to have something like this:
qname=“triatn.org.ua.”
qname_first=“ua.”
qname_second=“org.ua.”

I need to create a dashboard, where will be possible to get some statistics which related to dns usage.

2

I can’t think of an easy way to do this, support for mutating the json in promtail/agent doesn’t really exist as a first class option.

However, Promtail’s json parser uses JMESPath so there might be kind of a hacky way you could do something like this:

- json:
    expressions:
      new_output: "some crazy jmespath expression which appends the values you want to the existing object"
- output:
    new_output

Also be careful about extracting content into labels, the fields you have listed there are not things I would want as labels in the index. Try to keep labels to describe where your logs come from and not describing the content of your logs. It’s almost always better to use query time parsing to filter on values like this.

Check out the concise guide to labels and the Loki 2.0 blog posts to get an idea of how to do this with Loki for best performance .

1 Like