Dnssrv+ Not Working for scheduler_address in Loki Simple Scalable Mode

Grafana Loki
1

Hi all,

I’m trying to deploy Loki in Simple Scalable mode in Nomad, using Consul DNS for service discovery.

The dnssrv+ prefix works correctly in some parts of the config — for example:

  • common.compactor_address
  • memberlist.join_members

However, when I use dnssrv+ or dns+ for the following:

  • frontend.scheduler_address
  • frontend_worker.scheduler_address

it fails with:

caller=dns_resolver.go:225 msg="failed DNS A record lookup" err="lookup dnssrv+_loki-backend._grpc.service.dev.consul.: no such host"

To isolate the problem, I reproduced this setup using Docker Compose + Consul + CoreDNS, and the issue still persists.

Here’s a trimmed version of my loki-config.yml:

yaml

common:
  path_prefix: ${LOKI_DATA_PATH}
  replication_factor: 2
  ring:
    kvstore:
      store: memberlist
  compactor_address: dnssrv+_loki-backend._grpc.service.dev.consul.

frontend:
  scheduler_address: dnssrv+_loki-backend._grpc.service.dev.consul.
  grpc_client_config:
    tls_enabled: false

frontend_worker:
  scheduler_address: dnssrv+_loki-backend._grpc.service.dev.consul.
  grpc_client_config:
    tls_enabled: false

server:
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9096

I’ve verified that the SRV record resolves manually using dig, The query succeeds, and compactor_address works fine using the same format. So the DNS resolution itself is working. It appears that scheduler_address may not support dnssrv+ or may be parsing it differently.

Questions:

  • Is dnssrv+ supported in scheduler_address fields?
  • Is there any difference in how Loki internally resolves compactor_address vs scheduler_address?
  • Any recommended workaround?

Any guidance would be appreciated — thank you!

2

I could be wrong, but I believe it is a host:port connection (meaning it just establishes connection, no service discovery), because it only needs to connect to one of the schedulers, not all of them.

If you have dedicated IP and static port for gRPC for each of your container, then I think you can use service discovery name with the same static port. Otherwise you might need some sort of service endpoint (i don’t know if nomad has this).

3

Thanks, @tonyswumac — It ended up working after I removed the dnssrv+ prefix and just used the Consul DNS name directly in the scheduler_address. I didn’t need to use a static IP — just the regular Consul DNS name like:

scheduler_address: _loki-backend._grpc.service.dev.consul.:<port>

However, I did have to keep the gRPC port static to ensure it connected successfully.