Unable to set service_name in a Docker/Alloy/Loki setup

I’m trying to use the otel-lgtm docker image to locally test OpenTelemetry traces and logs correlation but I’ve been unable to correctly set the service_name label/resource attribute.
I have this compose.yml

services:
  logger:
    image: busybox
    command: sh -c 'while true; do echo "[$(date)] Infinite log message..."; sleep 5; done'
    depends_on:
      lgtm:
        condition: service_healthy
    labels:
      com.local.monitor-logs: "true"

  alloy:
    image: grafana/alloy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./config.alloy:/etc/alloy/config.alloy
    depends_on:
      lgtm:
        condition: service_healthy

  lgtm:
    image: grafana/otel-lgtm
    environment:
    - OTEL_METRIC_EXPORT_INTERVAL=500
    ports:
      - 5000:3000
    healthcheck:
      test: test -e /tmp/ready
      interval: 5s
      timeout: 2s
      retries: 20

With this alloy.config

discovery.docker "containers_with_logs" {
  host = "unix:///var/run/docker.sock"

  filter {
    name = "label"
    values = ["com.local.monitor-logs=true"]
  }
}

discovery.relabel "docker_compose_services" {
    targets = discovery.docker.containers_with_logs.targets

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_project"]
        target_label  = "project"
    }

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
        target_label  = "service"
    }

    rule {
        source_labels = ["__meta_docker_container_label_com_docker_compose_service"]
        target_label  = "service_name"
    }
}

loki.source.docker "docker_compose_services_logs" {
  host = "unix:///var/run/docker.sock"

  targets = discovery.docker.containers_with_logs.targets
  relabel_rules = discovery.relabel.docker_compose_services.rules
  labels = {"app" = "docker"}
  forward_to = [otelcol.receiver.loki.default.receiver]
}

otelcol.receiver.loki "default" {
  output {
    logs = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "lgtm:4317"
    tls {
      insecure = true
      insecure_skip_verify = true
    }
  }
}

Logs are ingested fine but I’m getting service_name=unknown_service and service_name_extracted=logger with every config option that allows me to set the service_name (or service.name)

I tried several other config options (Can't set logs service_name for docker logs · Issue #261 · grafana/docker-otel-lgtm · GitHub) but nothing seem to work.

Am I doing something wrong?

Emable live debugging and check log structure om the otel exporter level. There must be resource attribute service.name with meaningful value. You can use some processor to configure it somehow.

The otelcol.exporter.otlp.default doesn’t seem to support live debugging but if I check the otelcol.processor.attributes.default component (which comes right before the exporter) I see these kind of objects

ResourceLog #0
Resource SchemaURL: 
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope  
LogRecord #0
ObservedTimestamp: 2025-01-28 13:31:15.63021166 +0000 UTC
Timestamp: 2025-01-28 13:31:15.628681385 +0000 UTC
SeverityText: 
SeverityNumber: Unspecified(0)
Body: Str(INFO:     Stopping reloader process [1])
Attributes:
     -> loki.attribute.labels: Str(app,project,service_name)
     -> app: Str(docker)
     -> project: Str(core-service)
     -> service_name: Str(some-service)
     -> service.name: Str(abc)
     -> service: Str(xyz)
     -> other_attr: Str(def)
     -> discover_service_name: Bool(false)
Trace ID: 
Span ID: 
Flags: 0

As you can see, there are several attempts to set the service name but on Loki they still appear as service_name=unknown_service.

Is there some other way to debug the exporter? Or something else I can try?

1 Like

Those attributes are log attributes, not resource attributes.You can use transform processor to set resources attribute from log attribute, e.g.

set(resource.attributes["service.name"], attributes["service.name"])

Hmm I see, that makes sense. I couldn’t find any reference to “transfer processor” on alloy docs though, do you mind giving me some more reference about it?

I did try at some point setting the attribute using a loki.process component but it did not work either

loki.process "default" {
  stage.docker {}

  stage.labels {
    values = {
      service_name = "some-other-service",
      service_name_2 = "some-other-service",
    }
  }

  stage.static_labels {
    values = {
      service_name = "some-service",
    }
  }

  forward_to = [otelcol.receiver.loki.default.receiver]
}

This is the output of the livedebugging for the loki.process component

[IN]: timestamp: 2025-01-28T14:36:56.268045122Z, entry: INFO:     Stopping reloader process [1], labels: {app="docker", project="core-service", service_name="backend"}

It seems all labels and static_labels are ignored in this case (or at least they don’t show up in this component debug view; the static label does show up in other component debug info)

    otelcol.processor.transform "fixes" {
      error_mode = "silent"

      log_statements {
        context = "log"
        statements = [
          `set(resource.attributes["service.name"], attributes["service.name"])`,
        ]
      }

      output {
        logs    = [...]
      }
    }

Alloy is/was focused on Loki/Loki exporter and there is option to use log attributes to define Loki labels - you will need less configs for Loki.
But OTLP exporter needs transformations to achieve the same results.

I am having trouble setting the service_name in my Docker/Alloy/Loki setup. Has anyone faced this issue?