Alloy Configuration

Grafana Alloy
1

Hi guys, would like to ask, currently I have configured alloy for EKS to scrape logs from application pods, however, I am facing issue when I tried to setup index labels, it became duplicate label.

Screenshot 2025-05-29 at 7.20.22 PM
Screenshot 2025-05-29 at 7.20.22 PM1276Ă—270 37.2 KB

My Loki.process are as below

loki.process "pod_logs" {
  stage.static_labels {
    values = {
      cluster = "dev",
    }
  }

  stage.json {
    expressions = { 
      ts = "",
      level = "",
      traceID = "", 
      spanID = "",  
      path = "", 
      method = "", 
      status = "",
      msg = "",
    }
  }

  stage.labels {
    values = {
      traceID = "",
      spanID  = "",
      level   = "",
    }
  }

  stage.timestamp {
    source = "ts"
    format = "Unix"
  }

  forward_to = [loki.process.remove_labels.receiver]
}

Logs sample output are as below:

{"level":"info","ts":"2025-05-29T11:26:10Z","msg":"Request processed successfully","status_code":200,"request_method":"GET","request_url":"/api/v1/healthz","protocol":"HTTP/1.1","user_agent":"kube-probe/1.32+","ip_address":"<redacted>","referer":"","content_type":"","content_length":"","traceID":"b2df9bd9ece531e34d23b2d993d91be6","spanID":"ccfa182d327cc0a4"}

Any help would be appreciated,
Thank you

2

Are you trying to extract fields from your logs using empty expressions? It looks like in your stage.json, you’re assigning empty strings to variables. Instead, each variable should map to the actual key name from your JSON log. Here’s how it should look:

stage.json {
  expressions = { 
    ts = "ts",
    level = "level",
    traceID = "traceID",
    spanID = "spanID",
    status_code = "status_code",
    method = "request_method",
    msg = "msg",
  }
}

Also, please keep in mind that fields like traceID and timestamp have high cardinality, which can significantly impact Loki’s performance and storage efficiency, especially if you use them as labels.

More info: Loki Label Cardinality

1 Like
3

Hey @kreg870, thanks for the reply man. I followed the guide here, loki.process | Grafana Agent documentation, where it mentioned if assign = “” will result in using the same value as the variable.

Btw, thanks for the information on the high cardinality, will totally read more about this. However will hope to be able to resolve this issue as well. As I am using Jaegar and Loki while correlating them together, Loki can linked to Jaegar without any issue, but from Jaegar linking back to Loki, I am using Trace ID to find the appropriate logs, but due to the trace id not being part of the label, I can’t find it.

4

I managed to tune the Jaegar instead to be:

{service_name="${__span.tags['otel.scope.name']}"} |=`${__span.spanId}` |= `${__trace.traceId}`

and now it works for the filter part and no labels is require