- What Grafana version and what operating system are you using?
Grafana 11.4.0
Hello everyone, I’m having a little bit of trouble cutting down data from our logs. We sometimes had 10mb on data for one dashboard which creates a lot of overhead. Our plan was to cut down data from incoming logs by only filtering the necessary fields. Our biggest problem right now is that we still need all the data from time to time (mostly on error logs). We currently have a working setup by filtering necessary fields to our dashboard: | json | keep body, service_name, level, deployment_tenant, traceid, resources_host_name | line_format {{.body}} - “traceid”:“{{.traceid}}”``.
After that we add a label ‘time_in_ns’ to the query that adds “{{timestamp | unixEpochNanos}}” to the query results. After that we added a ‘Derived field’ to Loki Data sources so it refers to that specific label and adds a link to it so we can view it in Explore.
Now, it does work great (at the moment) but I’m very skeptical about the query’s we used for it. Here’s the query for the dashboard: {service_namespace="$namespace", level=~"$level", service_name=~"$servicename", deployment_environment=~"$environment", deployment_tenant=~"$tenant"} |= `` | json | keep body, service_name, level, deployment_tenant, traceid, resources_host_name | line_format {{.body}} - “traceid”:“{{.traceid}}” | label_format time_in_ns="{{__timestamp__ | unixEpochNanos}}"
And here’s the query for the Derived field (all variables are variables from our dashboard): /explore?left={"datasource":"loki","queries":[{"expr":"{service_namespace=\"$namespace\", deployment_tenant=~\"${tenant:pipe}\", service_name=~\"${servicename:pipe}\", deployment_environment=~\"${environment:pipe}\", level=~\"${level:pipe}\"} | json | label_format time_in_ns=\"{{__timestamp__ | unixEpochNanos}}\" | time_in_ns = ${__value.raw} | keep attributes_exception_stacktrace, body | line_format \"{{.body}} \\n\\n {{.attributes_exception_stacktrace}}\""}],"queryType":"range","range":{"from":"${__from}","to":"${__to}"}}
I was able to reduce the overhead from log data by half, but the query’s are overly complex. Is there any way to batch these results so we can keep it in one dashboard? Or are the any other, better ways to do this?