Custom labels using OpenTelemetry Collector and Loki

Grafana Loki
1

Hello everyone,
I’m trying to add some custom labels (hostname and application component) to Loki, which is configured as a Datasource in Grafana.
The data flux is like this:

Open Telemetry Agent on host (ex host 221) has receiver logs, exporter another Open Telemetry Collector (For simplicity, let’s call it Otel DC), this one has as receiver the Open Telemetry agent (localhost:port) and as exporter another Open Telemetry Collector (which is Otel Main). This main collector finally has as reciver multiple Otel Collectors and as exporter Loki.
So it’s like this:

Otel Agent → Otel DC → Otel Main → Loki

In Otel agent i added some informations that helps us identify the host, which are added using the following processor:

processors:
  resource:
    attributes:
    - key: deployment.environment
      value: "PROD"
      action: upsert
    - key: service.name
      value: "DHub"
      action: upsert
    - key: host.name
      value: "va221"
      action: upsert
    - key: application.component
      value: "DHub-Ex"
      action: upsert

We would like to use “host.name” and “application.component” as Labels in Loki like “service.name” and “deployment.environment”:

image
image1039×100 7.63 KB

We see them below as Common Labels:

image
image742×151 11.3 KB

We’re using the otlphttp exporter in Otel main to send the informations to Grafana Loki as follow:

exporters:
    otlphttp/prod_logs:
      endpoint: http://<loki_url>:3100/otlp
      headers:
        X-Scope-OrgID: "<tenant-1>"
      auth:
        authenticator: basicauth/client

Currently we’re using Chart versions 6.24.0.
Thanks to everyone in advice.

2

Configure your Loki and define, which resource attributes will be indexed as a Loki labels with default_resource_attributes_as_index_labels config option:

Example:

distributor:
  otlp_config:
    default_resource_attributes_as_index_labels:
      - "cloud.availability_zone"
      - "cloud.region"
      - "container.name"
      - "deployment.environment"
      - "k8s.cluster.name"
      - "k8s.container.name"
      - "k8s.cronjob.name"
      - "k8s.daemonset.name"
      - "k8s.deployment.name"
      - "k8s.job.name"
      - "k8s.namespace.name"
      - "k8s.node.name"
      - "k8s.pod.name"
      - "k8s.replicaset.name"
      - "k8s.statefulset.name"
      - "service.instance.id"
      - "service.name"
      - "service.namespace"
1 Like
3

Hello,
First of all, thanks for the reply!
Second thing, i tried to add it like this:

distributor:
  otlp_config:
    default_resource_attributes_as_index_labels:
      - "host.name"
      - "application.component"
      - "container.name"
      - "deployment.environment"
      - "k8s.cluster.name"
      - "k8s.container.name"
      - "k8s.cronjob.name"
      - "k8s.daemonset.name"
      - "k8s.deployment.name"
      - "k8s.job.name"
      - "k8s.namespace.name"
      - "k8s.node.name"
      - "k8s.pod.name"
      - "k8s.replicaset.name"
      - "k8s.statefulset.name"
      - "service.instance.id"
      - "service.name"
      - "service.namespace"

At the “base” level, like read/write/backend and, after restarting it, there was no changes sadly, tried inside the loki block as well, no changes again.

P.s. i can share the yaml if needed, and right now we’re using the deploymentMode as SimpleScalable

Best regards

4

Hello again!
Just to correct my last post, it was my fault that there was wrong indentation on the yaml.
It works if it’s inserted into the loki block, like this:

loki:
  distributor:
    otlp_config:
      default_resource_attributes_as_index_labels:
        - "host.name"
        - "application.component"
        - "container.name"
        - "deployment.environment"
        - "k8s.cluster.name"
        - "k8s.container.name"
        - "k8s.cronjob.name"
        - "k8s.daemonset.name"
        - "k8s.deployment.name"
        - "k8s.job.name"
        - "k8s.namespace.name"
        - "k8s.node.name"
        - "k8s.pod.name"
        - "k8s.replicaset.name"
        - "k8s.statefulset.name"
        - "service.instance.id"
        - "service.name"
        - "service.namespace"

Is there a limit of these static lables, or i can add as many as i want?

5
6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.