schmoho
December 14, 2022, 8:46am
1
My Loki logs are producing lots and lots of “context canceled” errors which correlates to my running queries (through Grafana) quite nicely.
Also note that the queries are executed without apparent problems and return logs alright.
However, I don’t really understand the error messages (pretty much a noob here).
My setup so far is mostly for testing purposes, so it is local (using the 2.7.1 Docker image) and not scaled whatsoever.
There seems to be a related issue which was reported here:
opened 04:57PM - 09 Nov 22 UTC
Hi,
- Docker version 20.10.17, build 100c701
- Docker Compose version v2.10.… 2
- Ubuntu WSL2
- loki:2.6.1
I am getting a lot of those errors:
```
loki-1 | level=error ts=2022-11-09T16:43:52.196413829Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
loki-1 | level=error ts=2022-11-09T16:43:52.196413176Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
loki-1 | level=error ts=2022-11-09T16:43:52.196428849Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
loki-1 | level=error ts=2022-11-09T16:43:52.196462256Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
loki-1 | level=error ts=2022-11-09T16:43:52.196970284Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=0.0.0.0:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197027425Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197058086Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=0.0.0.0:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197084426Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197793087Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=0.0.0.0:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197823878Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=0.0.0.0:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197833438Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197837788Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197860028Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=0.0.0.0:9096
loki-1 | level=error ts=2022-11-09T16:43:52.197879088Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=127.0.0.1:9096
```

I found similar issue about related to frontend address, I added it to config, but didnt help
```
frontend:
address: 0.0.0.0
```
Here is my docker-compose.yml
```yml
loki:
image: grafana/loki:2.6.1
user: root
volumes:
- ./Docker/compose/local/loki:/etc/loki
- loki_data:/home/loki/data
ports:
- 3100:3100
- 9096:9096
restart: unless-stopped
command: -config.file=/etc/loki/loki-config.yml
```
loki-config.yml based on complete-local-config.yaml from Docs
```yml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: warn
frontend:
address: 0.0.0.0
common:
path_prefix: /home/loki/data
storage:
filesystem:
chunks_directory: /home/loki/data/chunks
rules_directory: /home/loki/data/rules
replication_factor: 1
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
query_scheduler:
max_outstanding_requests_per_tenant: 1000
```
Any ideas what is causing it?
Thank you
Here is my config:
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
compactor:
retention_enabled: true
delete_request_cancel_period: 15m
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
And these are the error messages in question:
level=error ts=2022-12-14T08:34:53.836753189Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=<loki-container-ip-address>:9095
level=error ts=2022-12-14T08:34:53.836740365Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=<loki-container-ip-address>:9095
level=error ts=2022-12-14T08:34:53.783633288Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.76464648Z caller=scheduler_processor.go:137 org_id=fake msg="error notifying scheduler about finished query" err=EOF addr=<loki-container-ip-address>:9095
level=error ts=2022-12-14T08:33:40.764637123Z caller=scheduler_processor.go:182 org_id=fake msg="error notifying frontend about finished query" err="rpc error: code = Canceled desc = context canceled" frontend=<loki-container-ip-address>:9095
level=error ts=2022-12-14T08:33:40.708772286Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708748542Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.70873676Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708736189Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708733935Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708732643Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708732392Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
level=error ts=2022-12-14T08:33:40.708728806Z caller=retry.go:73 org_id=fake msg="error processing request" try=0 err="context canceled"
I have been trying to understand what is happening here, but have not made any real progress. So if anyone could help me understand I would be much obliged!
Same problem in here.
I’m using chart Loki-distributed 0.69.1 and Loki 2.7.1