Help to “translate” an “egrep|jq” combo to LogQL

Grafana Loki
,
1

Dear Forum Members.

I’m new to loki and LogQL and want to “translate” this “egrep|jq” combo to LogQL.

oc -n kube-ns-dev3 logs caddy-67545f989c-gnsk8 |\
egrep -v "metrics|debug" | \
egrep 'logger":"http.log.access' |\
jq '.request.uri,.duration'

The first grep filters the /metrics and the debug level message, the second grep selects only the access logs because there are also some TLS logs in the output.

My current Query is this, which cretes the fields request_uri and duration.

{ log_type="application", kubernetes_pod_name =~"caddy-.+"} |="duration" | json | line_format "{{.message}}" | json

That’s the log line.

{"level":"info","ts":"2025-02-25T13:24:31.005Z","logger":"http.log.access","msg":"handled request","request":{"remote_ip":"10.131.0.21","remote_port":"41610","client_ip":"217.149.233.32","proto":"HTTP/1.1","method":"GET","host":"dev3.dev.internal.domain","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"Sec-Ch-Ua":["\"Chromium\";v=\"122\", \"Not(A:Brand\";v=\"24\", \"Google Chrome\";v=\"122\""],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Port":["443"],"Forwarded":["for=10.196.15.7;host=dev3.dev.internal.domain;proto=https"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br, zstd"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36"],"X-Original-Url":["/"],"X-Original-Host":["dev3.dev.internal.domain"],"X-Forwarded-Proto":["https"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Ch-Ua-Platform":["\"Windows\""],"X-Forwarded-Host":["dev3.dev.internal.domain"],"Cookie":["REDACTED"],"Sec-Fetch-User":["?1"]}},"bytes_read":0,"user_id":"","duration":2681,"size":6271,"status":200,"resp_headers":{"Server":["Caddy"],"X-Content-Type-Options":["nosniff"],"Vary":["Accept-Encoding"],"Referrer-Policy":["no-referrer-when-downgrade"],"X-Frame-Options":["SAMEORIGIN"],"X-Robots-Tag":["none"],"Cache-Control":["public, s-maxage=31536000, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Xss-Protection":["1; mode=block"],"Content-Encoding":["br"],"Permissions-Policy":["interest-cohort=()"],"Strict-Transport-Security":["max-age=31536000; includeSubDomains; preload"]}}

How can I now create a view to see the duration per request_uri?

Thanks for any help

Regards

Alex

2

First of all, your log is not valid JSON. Ignoring the obvious windows-styled quote (assuming that’s just from copy pasting), this part is certainly not JSON:

"Sec-Ch-Ua":[""Chromium";v="122", "Not(A:Brand";v="24", "Google Chrome";v="122""]

If you cannot fix it and make it a valid JSON string, then your best bet might just be to search for duration and regex both duration and URI, Something like:

{<SELECTOR>} |~ `(?i)duration` | regexp `.+\"duration\"\:(?P<duration>\d+)` | regexp `.+\"uri\"\:\"(?P<uri>[^\"]+)\"`

Not ideal for sure. I’d recommend fixing log to valid json format if possible.

1 Like
3

Well it’s a valid json in raw text.

Looks like the formatting of the forum removes the \".
I have now uses the backtick raw formatting in the original request instead the > nicer looking formatting.

Sorry for the inconvenience.

4

I see. Then you should be able to use JSON and have all the labels processed for you. To calculate average session time aggregated by URI, try this (not tested):

sum by (uri) (
  avg_over_time({SELECTOR}
    | json
    | unwrap duration
    [$__auto]
  )
)
1 Like
5

Thank you for the suggestion.

Maybe there is a tip how to reproduce the egrep -v "metrics|debug" part in loki because I have tried != but I still have the metrics in the output?

I’m not sure if I should use the " Line filter expression" or the " Log stream selector"

{ log_type="application", kubernetes_pod_name =~"caddy-.+"} != "metrics" | json | line_format "{{.message}}" | json
6

Try |~ "(metrics|debug)"

1 Like