Hello,
I am attempting to connect Grafana to use Loki as a data source.
I am using the Loki-distributed helm chart in a k8s cluster. As far as i can tell, everything is working on write side of the pipeline. The only issue I have is connecting to the read part of the process.
grafana ui returns the following error:
Loki: Internal Server Error. 500. rpc error: code = Unavailable desc = name resolver error: produced zero addresses
I input the url loki-loki-distributed-query-frontend:3100 into the HTTP URL field and receive the following logs in debug mode:
k logs pod/loki-loki-distributed-querier-0
2021-12-17 20:39:26.076078 I | proto: duplicate proto type registered: purgeplan.DeletePlan
2021-12-17 20:39:26.076127 I | proto: duplicate proto type registered: purgeplan.ChunksGroup
2021-12-17 20:39:26.076133 I | proto: duplicate proto type registered: purgeplan.ChunkDetails
2021-12-17 20:39:26.076137 I | proto: duplicate proto type registered: purgeplan.Interval
2021-12-17 20:39:26.098305 I | proto: duplicate proto type registered: grpc.PutChunksRequest
2021-12-17 20:39:26.098322 I | proto: duplicate proto type registered: grpc.GetChunksRequest
2021-12-17 20:39:26.098325 I | proto: duplicate proto type registered: grpc.GetChunksResponse
2021-12-17 20:39:26.098328 I | proto: duplicate proto type registered: grpc.Chunk
2021-12-17 20:39:26.098330 I | proto: duplicate proto type registered: grpc.ChunkID
2021-12-17 20:39:26.098333 I | proto: duplicate proto type registered: grpc.DeleteTableRequest
2021-12-17 20:39:26.098336 I | proto: duplicate proto type registered: grpc.DescribeTableRequest
2021-12-17 20:39:26.098339 I | proto: duplicate proto type registered: grpc.WriteBatch
2021-12-17 20:39:26.098342 I | proto: duplicate proto type registered: grpc.WriteIndexRequest
2021-12-17 20:39:26.098345 I | proto: duplicate proto type registered: grpc.DeleteIndexRequest
2021-12-17 20:39:26.098348 I | proto: duplicate proto type registered: grpc.QueryIndexResponse
2021-12-17 20:39:26.098350 I | proto: duplicate proto type registered: grpc.Row
2021-12-17 20:39:26.098353 I | proto: duplicate proto type registered: grpc.IndexEntry
2021-12-17 20:39:26.098355 I | proto: duplicate proto type registered: grpc.QueryIndexRequest
2021-12-17 20:39:26.098358 I | proto: duplicate proto type registered: grpc.UpdateTableRequest
2021-12-17 20:39:26.098361 I | proto: duplicate proto type registered: grpc.DescribeTableResponse
2021-12-17 20:39:26.098363 I | proto: duplicate proto type registered: grpc.CreateTableRequest
2021-12-17 20:39:26.098366 I | proto: duplicate proto type registered: grpc.TableDesc
2021-12-17 20:39:26.098368 I | proto: duplicate proto type registered: grpc.TableDesc.TagsEntry
2021-12-17 20:39:26.098371 I | proto: duplicate proto type registered: grpc.ListTablesResponse
2021-12-17 20:39:26.098373 I | proto: duplicate proto type registered: grpc.Labels
2021-12-17 20:39:26.098492 I | proto: duplicate proto type registered: storage.Entry
2021-12-17 20:39:26.098503 I | proto: duplicate proto type registered: storage.ReadBatch
level=info ts=2021-12-17T20:39:26.104424711Z caller=main.go:94 msg="Starting Loki" version="(version=2.4.1, branch=HEAD, revision=f61a4d261)"
level=info ts=2021-12-17T20:39:26.104742789Z caller=server.go:260 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2021-12-17T20:39:26.105408266Z caller=memberlist_client.go:394 msg="Using memberlist cluster node name" name=loki-loki-distributed-querier-0-bd677d02
ts=2021-12-17T20:39:26.105515764Z caller=memberlist_logger.go:74 level=debug msg="configured Transport is not a NodeAwareTransport and some features may not work as desired"
level=debug ts=2021-12-17T20:39:26.106076551Z caller=tcp_transport.go:389 msg=FinalAdvertiseAddr advertiseAddr=192.168.80.3 advertisePort=7946
level=debug ts=2021-12-17T20:39:26.106623651Z caller=tcp_transport.go:389 msg=FinalAdvertiseAddr advertiseAddr=192.168.80.3 advertisePort=7946
ts=2021-12-17T20:39:31.111316216Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 192.168.80.5:7946"
ts=2021-12-17T20:39:31.118814279Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 192.168.104.1:7946"
ts=2021-12-17T20:39:31.125709539Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 192.168.40.4:7946"
ts=2021-12-17T20:39:31.129798684Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 192.168.0.5:7946"
level=debug ts=2021-12-17T20:39:31.130616164Z caller=worker_service.go:149 msg="determining if querier is running as standalone target" runningStandalone=true queryFrontendEnabled=false queryScheduleEnabled=false readEnabled=false allEnabled=false
level=info ts=2021-12-17T20:39:31.130969834Z caller=worker.go:118 msg="Starting querier worker connected to query-frontend" frontend=loki-loki-distributed-query-frontend:9095
level=info ts=2021-12-17T20:39:31.132122215Z caller=module_service.go:64 msg=initialising module=server
level=debug ts=2021-12-17T20:39:31.132159294Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=ingester-querier
level=debug ts=2021-12-17T20:39:31.132209577Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=memberlist-kv
level=debug ts=2021-12-17T20:39:31.132200668Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=ingester-querier
level=debug ts=2021-12-17T20:39:31.132275534Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=memberlist-kv
level=info ts=2021-12-17T20:39:31.132281669Z caller=module_service.go:64 msg=initialising module=memberlist-kv
level=debug ts=2021-12-17T20:39:31.132377902Z caller=module_service.go:54 msg="module waiting for initialization" module=ring waiting_for=server
level=info ts=2021-12-17T20:39:31.132394615Z caller=module_service.go:64 msg=initialising module=ring
level=debug ts=2021-12-17T20:39:31.133995333Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=ring
ts=2021-12-17T20:39:31.135239045Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: 192.168.40.5:7946"
level=debug ts=2021-12-17T20:39:31.135334494Z caller=module_service.go:54 msg="module waiting for initialization" module=ingester-querier waiting_for=server
level=info ts=2021-12-17T20:39:31.135437402Z caller=module_service.go:64 msg=initialising module=ingester-querier
level=debug ts=2021-12-17T20:39:31.135645783Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=memberlist-kv
level=debug ts=2021-12-17T20:39:31.135763797Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=ring
level=debug ts=2021-12-17T20:39:31.135864266Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=server
level=debug ts=2021-12-17T20:39:31.135910366Z caller=module_service.go:54 msg="module waiting for initialization" module=querier waiting_for=store
level=debug ts=2021-12-17T20:39:31.135947607Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=memberlist-kv
level=debug ts=2021-12-17T20:39:31.135998775Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=ring
level=debug ts=2021-12-17T20:39:31.136044975Z caller=module_service.go:54 msg="module waiting for initialization" module=store waiting_for=server
level=info ts=2021-12-17T20:39:31.136093662Z caller=module_service.go:64 msg=initialising module=store
level=info ts=2021-12-17T20:39:31.136162376Z caller=module_service.go:64 msg=initialising module=querier
level=info ts=2021-12-17T20:39:31.136841905Z caller=loki.go:318 msg="Loki started"
level=info ts=2021-12-17T20:39:31.137552251Z caller=memberlist_client.go:506 msg="joined memberlist cluster" reached_nodes=5
level=info ts=2021-12-17T20:39:31.14032515Z caller=worker.go:205 msg="adding connection" addr=192.168.0.11:9095
level=warn ts=2021-12-17T20:39:31.140556783Z caller=worker.go:268 msg="total worker concurrency is greater than promql max concurrency. Queries may be queued in the querier which reduces QOS"
level=debug ts=2021-12-17T20:40:01.635599626Z caller=logging.go:67 traceID=2b1428317ab3a623 orgID=fake msg="GET /ready (200) 173.496µs"
ts=2021-12-17T20:40:03.95294199Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: loki-loki-distributed-distributor-7745846956-fntjp-c830ed9b 192.168.80.5:7946"
level=debug ts=2021-12-17T20:40:11.635458034Z caller=logging.go:67 traceID=0752abf101843d53 orgID=fake msg="GET /ready (200) 91.401µs"
level=debug ts=2021-12-17T20:40:21.635386767Z caller=logging.go:67 traceID=6a5c91e34c70de62 orgID=fake msg="GET /ready (200) 85.358µs"
level=debug ts=2021-12-17T20:40:31.635313832Z caller=logging.go:67 traceID=067ae958c96da13c orgID=fake msg="GET /ready (200) 92.163µs"
ts=2021-12-17T20:40:33.956127909Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: loki-loki-distributed-ingester-1-c5c5835d 192.168.80.4:7946"
ts=2021-12-17T20:40:38.203226764Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.40.5:47758"
level=debug ts=2021-12-17T20:40:41.635377583Z caller=logging.go:67 traceID=2128778068f8c388 orgID=fake msg="GET /ready (200) 98.474µs"
ts=2021-12-17T20:40:46.011753182Z caller=memberlist_logger.go:74 level=debug msg="Stream connection from=192.168.40.4:54240"
level=debug ts=2021-12-17T20:40:51.636167044Z caller=logging.go:67 traceID=470f64e08e1c45d3 orgID=fake msg="GET /ready (200) 107.945µs"
level=debug ts=2021-12-17T20:41:01.635570039Z caller=logging.go:67 traceID=3396057c1584c79d orgID=fake msg="GET /ready (200) 110.94µs"
ts=2021-12-17T20:41:03.96657786Z caller=memberlist_logger.go:74 level=debug msg="Initiating push/pull sync with: loki-loki-distributed-compactor-5fbbbc57cf-zjpmh-18588d5f 192.168.104.2:7946"
level=debug ts=2021-12-17T20:41:11.635349911Z caller=logging.go:67 traceID=11294384adbd1cf8 orgID=fake msg="GET /ready (200) 61.693µs"
without debug enabled:
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=error ts=2021-12-17T18:24:35.278379163Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki2/loki-loki-distributed-querier-0[loki]: level=error ts=2021-12-17T18:24:35.278090494Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="rpc error: code = Unavailable desc = name resolver error: produced zero addresses"
loki2/loki-loki-distributed-querier-0[loki]: level=warn ts=2021-12-17T18:24:35.278178895Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/labels?end=1639765475270116729&start=1639764874731000000 (500) 4.689324ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3aed26e94f67384e:034b0791ba0afd6b:415e5d90759cb78b:0; "
loki2/loki-loki-distributed-querier-0[loki]: level=error ts=2021-12-17T18:24:35.30565576Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="rpc error: code = Unavailable desc = name resolver error: produced zero addresses"
loki2/loki-loki-distributed-querier-0[loki]: level=warn ts=2021-12-17T18:24:35.305831754Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/labels?end=1639765475270116729&start=1639764874731000000 (500) 27.118854ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3aed26e94f67384e:034b0791ba0afd6b:415e5d90759cb78b:0; "
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=error ts=2021-12-17T18:24:35.306224589Z caller=retry.go:73 org_id=fake msg="error processing request" try=1 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki2/loki-loki-distributed-querier-0[loki]: level=error ts=2021-12-17T18:24:35.309880753Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="rpc error: code = Unavailable desc = name resolver error: produced zero addresses"
loki2/loki-loki-distributed-querier-0[loki]: level=warn ts=2021-12-17T18:24:35.309966476Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/labels?end=1639765475270116729&start=1639764874731000000 (500) 3.327181ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3aed26e94f67384e:034b0791ba0afd6b:415e5d90759cb78b:0; "
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=error ts=2021-12-17T18:24:35.31026562Z caller=retry.go:73 org_id=fake msg="error processing request" try=2 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki2/loki-loki-distributed-querier-0[loki]: level=error ts=2021-12-17T18:24:35.313429444Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="rpc error: code = Unavailable desc = name resolver error: produced zero addresses"
loki2/loki-loki-distributed-querier-0[loki]: level=warn ts=2021-12-17T18:24:35.313517291Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/labels?end=1639765475270116729&start=1639764874731000000 (500) 2.83642ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3aed26e94f67384e:034b0791ba0afd6b:415e5d90759cb78b:0; "
loki2/loki-loki-distributed-querier-0[loki]: level=error ts=2021-12-17T18:24:35.316181803Z caller=chunk_store.go:524 org_id=fake msg="error querying storage" err="rpc error: code = Unavailable desc = name resolver error: produced zero addresses"
loki2/loki-loki-distributed-querier-0[loki]: level=warn ts=2021-12-17T18:24:35.31627526Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/labels?end=1639765475270116729&start=1639764874731000000 (500) 2.31158ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; X-Scope-Orgid: fake; uber-trace-id: 3aed26e94f67384e:034b0791ba0afd6b:415e5d90759cb78b:0; "
loki2/grafana-678f986c7-qmnkb[grafana]: t=2021-12-17T18:24:35+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/2/loki/api/v1/label status=500 remote_addr=10.220.115.238 time_ms=50 size=82 referer=https://grafana-icboca.risk.regn.net/datasources/edit/kk0KbSo7z
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=error ts=2021-12-17T18:24:35.313710476Z caller=retry.go:73 org_id=fake msg="error processing request" try=3 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=error ts=2021-12-17T18:24:35.316489147Z caller=retry.go:73 org_id=fake msg="error processing request" try=4 err="rpc error: code = Code(500) desc = rpc error: code = Unavailable desc = name resolver error: produced zero addresses\n"
loki2/loki-loki-distributed-query-frontend-c947c647b-zm82n[loki]: level=warn ts=2021-12-17T18:24:35.316576782Z caller=logging.go:72 traceID=3aed26e94f67384e orgID=fake msg="GET /loki/api/v1/label?start=1639764874731000000 (500) 46.538652ms Response: \"rpc error: code = Unavailable desc = name resolver error: produced zero addresses\\n\" ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.5; Dnt: 1; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin; Sec-Gpc: 1; User-Agent: Grafana/8.3.0; X-Forwarded-For: 10.220.115.238, 192.168.80.0, 192.168.80.0; X-Grafana-Nocache: true; X-Grafana-Org-Id: 1; X-Real-Ip: 10.220.115.238; X-Request-Id: 97b32aaa89d01331794683097835af95; X-Scheme: https;
Here is my config file.
auth_enabled: false
server:
http_listen_port: 3100
log_level: debug
distributor:
ring:
kvstore:
store: memberlist
memberlist:
join_members:
- {{ include "loki.fullname" . }}-memberlist
ingester:
lifecycler:
ring:
kvstore:
store: memberlist
replication_factor: 3
chunk_idle_period: 1h
chunk_block_size: 1536000
chunk_encoding: snappy
chunk_retain_period: 1m
max_transfer_retries: 0
wal:
dir: /var/loki/wal
limits_config:
ingestion_rate_strategy: local
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
ingestion_rate_mb: 15
ingestion_burst_size_mb: 20
per_stream_rate_limit: 10MB
schema_config:
configs:
- from: 2020-09-07
store: boltdb-shipper
object_store: aws
schema: v11
index:
prefix: loki_index_
period: 24h
storage_config:
aws:
s3: s3://xxxx:8080/lokibucket
s3forcepathstyle: true
index_queries_cache_config:
memcached:
batch_size: 100
parallelism: 100
memcached_client:
consistent_hash: true
host: {{ include "loki.memcachedIndexQueriesFullname" . }}
service: http
boltdb_shipper:
active_index_directory: /var/loki/index
shared_store: s3
cache_location: /var/loki/cache
index_gateway_client:
server_address: dns:///{{ include "loki.indexGatewayFullname" . }}:9095
chunk_store_config:
chunk_cache_config:
memcached:
batch_size: 100
parallelism: 100
memcached_client:
consistent_hash: true
host: {{ include "loki.memcachedChunksFullname" . }}
service: http
table_manager:
retention_deletes_enabled: false
retention_period: 0s
querier:
query_store_only: true
query_range:
# make queries more cache-able by aligning them with their step intervals
align_queries_with_step: true
max_retries: 5
# parallelize queries in 15min intervals
split_queries_by_interval: 15m
cache_results: true
results_cache:
cache:
# We're going to use the in-process "FIFO" cache
enable_fifocache: true
fifocache:
size: 1024
validity: 24h
frontend_worker:
frontend_address: {{ include "loki.queryFrontendFullname" . }}:9095
frontend:
log_queries_longer_than: 5s
compress_responses: true
tail_proxy_url: http://{{ include "loki.querierFullname" . }}:3100
compactor:
working_directory: /loki/boltdb-shipper-compactor
shared_store: s3
ruler:
storage:
type: local
local:
directory: /etc/loki/rules
ring:
kvstore:
store: memberlist
rule_path: /tmp/loki/scratch
alertmanager_url: https://alertmanager.xx
external_url: https://alertmanager.xx
k get pods
loki-loki-distributed-compactor-547979c8d8-rtbr7 1/1 Running 0 10d
loki-loki-distributed-distributor-55b4f64f9-55g5p 1/1 Running 0 10d
loki-loki-distributed-distributor-55b4f64f9-59th7 1/1 Running 0 10d
loki-loki-distributed-distributor-55b4f64f9-jb2n4 1/1 Running 0 10d
loki-loki-distributed-gateway-7896dbfb9-hb292 1/1 Running 0 12d
loki-loki-distributed-gateway-7896dbfb9-j6xvr 1/1 Running 0 12d
loki-loki-distributed-gateway-7896dbfb9-v5z7l 1/1 Running 0 12d
loki-loki-distributed-ingester-0 1/1 Running 0 10d
loki-loki-distributed-ingester-1 1/1 Running 0 10d
loki-loki-distributed-ingester-2 1/1 Running 0 10d
loki-loki-distributed-memcached-chunks-0 2/2 Running 0 12d
loki-loki-distributed-memcached-chunks-1 2/2 Running 0 12d
loki-loki-distributed-memcached-chunks-2 2/2 Running 0 12d
loki-loki-distributed-memcached-chunks-3 2/2 Running 0 12d
loki-loki-distributed-memcached-chunks-4 0/2 Pending 0 12d
loki-loki-distributed-memcached-frontend-0 2/2 Running 0 12d
loki-loki-distributed-memcached-frontend-1 2/2 Running 0 12d
loki-loki-distributed-memcached-frontend-2 2/2 Running 0 12d
loki-loki-distributed-memcached-index-queries-0 2/2 Running 0 10d
loki-loki-distributed-memcached-index-queries-1 2/2 Running 0 10d
loki-loki-distributed-memcached-index-queries-2 2/2 Running 0 10d
loki-loki-distributed-memcached-index-writes-0 2/2 Running 0 12d
loki-loki-distributed-memcached-index-writes-1 2/2 Running 0 12d
loki-loki-distributed-memcached-index-writes-2 2/2 Running 0 12d
loki-loki-distributed-querier-0 1/1 Running 0 10d
loki-loki-distributed-querier-1 1/1 Running 0 10d
loki-loki-distributed-query-frontend-c947c647b-zm82n 1/1 Running 0 10d
k get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana ClusterIP 192.168.163.89 <none> 80/TCP 12d
loki-loki-distributed-compactor ClusterIP 192.168.186.160 <none> 3100/TCP 12d
loki-loki-distributed-distributor ClusterIP 192.168.195.44 <none> 3100/TCP,9095/TCP 12d
loki-loki-distributed-gateway ClusterIP 192.168.159.20 <none> 80/TCP 12d
loki-loki-distributed-ingester ClusterIP 192.168.168.177 <none> 3100/TCP,9095/TCP 12d
loki-loki-distributed-ingester-headless ClusterIP None <none> 3100/TCP,9095/TCP 12d
loki-loki-distributed-memberlist ClusterIP None <none> 7946/TCP 12d
loki-loki-distributed-memcached-chunks ClusterIP None <none> 11211/TCP,9150/TCP 12d
loki-loki-distributed-memcached-frontend ClusterIP None <none> 11211/TCP,9150/TCP 12d
loki-loki-distributed-memcached-index-queries ClusterIP None <none> 11211/TCP,9150/TCP 10d
loki-loki-distributed-memcached-index-writes ClusterIP None <none> 11211/TCP,9150/TCP 12d
loki-loki-distributed-querier ClusterIP 192.168.165.99 <none> 3100/TCP,9095/TCP 12d
loki-loki-distributed-querier-headless ClusterIP None <none> 3100/TCP,9095/TCP 12d
loki-loki-distributed-query-frontend ClusterIP None <none> 3100/TCP,9095/TCP,9096/TCP 12d
k get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
grafana <none> grafana-xxx.net 10.xxx,10.xxx,10.xxx,10.xxx 80, 443 11d
loki-loki-distributed-gateway <none> loki-xxx.net 10.xxx,10.xxx,10.xxx,10.xxx 80, 443 12d
The following logs message was of particular interest to me.
level=debug ts=2021-12-17T20:39:31.130616164Z caller=worker_service.go:149 msg="determining if querier is running as standalone target" runningStandalone=true queryFrontendEnabled=false queryScheduleEnabled=false readEnabled=false allEnabled=false