Hello. I am running loki v3.3.0 and ingesting logs from the local file system. I am noticing that I am getting constant messages with the text deadline exceeded. I am pretty sure that it is related to timeout that is too short but I have been unable to identify the correct setting to use in the loki yaml config. Any help would be greatly appreciated. Thank you.
Check your memory and CPU load.
Hi @tonyswumac. Thank you for the response. The machine that is running Loki has 32 CPUs and 64GB of memory. I just checked and do not see anything that raises a red flag. Is there anything specific you suggest I look for? Is there any Loki YAML config setting that could modify this observed behavior?
Usually deadline means some sort of resource contention.
Can you share your entire Loki configuration, please? What are the symptoms you’ve noticed so far besides deadline in the logs?
Hi @tonyswumac Other than the flood of messages appearing syslog, I am not noticing anything else out of the ordinary. I have noticed that there is a slight delay in log retrieval and I suspecting that is due to the high number deadline exceeded messages. The following is the Loki config YAML I am using:
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
log_level: debug
grpc_server_max_concurrent_streams: 1000
common:
instance_addr: 127.0.0.1
path_prefix: /tmp/loki
storage:
filesystem:
chunks_directory: /tmp/loki/chunks
rules_directory: /tmp/loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100
schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
pattern_ingester:
enabled: true
metric_aggregation:
loki_address: localhost:3100
ruler:
alertmanager_url: http://localhost:9093
frontend:
encoding: protobuf
compactor:
working_directory: /tmp/loki
compaction_interval: 10m
retention_enabled: true
retention_delete_delay: 2h
retention_delete_worker_count: 150
delete_request_store: filesystem
limits_config:
retention_period: 168h
ingestion_rate_mb: 8
ingestion_burst_size_mb: 12
ingester_client:
remote_timeout: 30m
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/analytics/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
# reporting_enabled: false
@tonyswumac Have you had a chance to take a look at my loki.yaml and if so does anything jump out that is wrong or missing?
Thank you.
Couple of things I would recommend:
- Try adding a couple of settings to the server block:
server:
# 100MB
grpc_server_max_recv_msg_size: 1.048576e+08
grpc_server_max_send_msg_size: 1.048576e+08
http_server_read_timeout: 600s
- Try changing compaction period to a bigger value. 1h should be minimum, 2h is ok too. You don’t need to compact frequently.
compactor:
compaction_interval: 2h # or 1h
Hi. I made the changes you suggested. I then restarted loki and also did a stop and start. I am still seeing the deadline messages coming in about once every second. I am assuming that the changes should take effect immediately, is that true. Any other settings or perhaps extending any of the timers?
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.335375534Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.406973342Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.406992712Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407001613Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407006853Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407011683Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407015563Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407019903Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407037033Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407066114Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:28 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:28.407121065Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.33790182Z caller=push.go:162 org_id=fake traceID=27d9c64fe6d83f3e msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.4 kB" streams=2 entries=14 streamLabelsSize="118 B" entriesSize="2.7 kB" structuredMetadataSize="0 B" totalSize="2.9 kB" mostRecentLagMs=790
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407374172Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407420413Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407433393Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407451844Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407457344Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407461514Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407465764Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407469724Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407473814Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:29 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:29.407477304Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.334622167Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=2
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.407947472Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.407975363Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.407985273Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.407992453Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.407998363Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.408004584Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.408010164Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.408017034Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.408022584Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.408028494Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:30 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:30.638428322Z caller=push.go:162 org_id=fake traceID=6d44605dad172cef msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.4 kB" streams=1 entries=22 streamLabelsSize="60 B" entriesSize="3.8 kB" structuredMetadataSize="0 B" totalSize="3.8 kB" mostRecentLagMs=85
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.334871812Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408036555Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408058045Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408068495Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408075625Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408082276Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408088236Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408093936Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408099356Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408105056Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.408110286Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:31 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:31.838450094Z caller=push.go:162 org_id=fake traceID=1c3400d5c4ccdbaa msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="987 B" streams=1 entries=12 streamLabelsSize="60 B" entriesSize="2.2 kB" structuredMetadataSize="0 B" totalSize="2.3 kB" mostRecentLagMs=283
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.335549319Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408859095Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408883746Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408893276Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408912796Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408919077Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408931197Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408936907Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408942867Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408947737Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:32 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:32.408952717Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.138289272Z caller=push.go:162 org_id=fake traceID=7417a908442a1745 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.0 kB" streams=1 entries=12 streamLabelsSize="60 B" entriesSize="2.2 kB" structuredMetadataSize="0 B" totalSize="2.3 kB" mostRecentLagMs=581
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.335528626Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409738791Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409758511Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409776221Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409806522Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409818132Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409824762Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409830103Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409835383Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409850923Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:33 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:33.409866623Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.338810776Z caller=push.go:162 org_id=fake traceID=6d1ba9b2631a1671 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.0 kB" streams=1 entries=12 streamLabelsSize="60 B" entriesSize="2.2 kB" structuredMetadataSize="0 B" totalSize="2.3 kB" mostRecentLagMs=779
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410175271Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410194531Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410202981Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410207681Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410212301Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410216291Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410220522Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410240002Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410244772Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:34 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:34.410248402Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.334900285Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410793477Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410809977Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=69
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410819008Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410826598Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=68
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410834078Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410840298Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=67
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410844768Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410850088Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=70
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410854608Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.410868209Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=71
Dec 4 14:18:35 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:35.6388691Z caller=push.go:162 org_id=fake traceID=40549dd9428fe519 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.3 kB" streams=1 entries=22 streamLabelsSize="60 B" entriesSize="3.8 kB" structuredMetadataSize="0 B" totalSize="3.8 kB" mostRecentLagMs=74
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334226308Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=0
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334244339Z caller=mock.go:196 msg=Get key=collectors/scheduler modify_index=70 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334267109Z caller=mock.go:123 msg=CAS key=collectors/scheduler modify_index=70 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.3342914Z caller=mock.go:196 msg=Get key=collectors/scheduler modify_index=72 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.33430649Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334417082Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=0
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334470513Z caller=mock.go:196 msg=Get key=collectors/compactor modify_index=69 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334522284Z caller=mock.go:123 msg=CAS key=collectors/compactor modify_index=69 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334662528Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.33475687Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=0
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334920554Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=0
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.334584376Z caller=mock.go:196 msg=Get key=collectors/compactor modify_index=73 value="\"G\\xf0F\\nE\\n\\x0eprimary-grover\\x123\\n\\x0e127.0.0.1:9096\\x10\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335290442Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335307322Z caller=mock.go:196 msg=Get key=collectors/pattern-ring modify_index=67 value="\"\\xbb\\x05\\xf4\\xba\\x02\\n\\xb8\\x05\\n\\x0eprimary-grover\\x12\\xa5\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335336982Z caller=mock.go:123 msg=CAS key=collectors/pattern-ring modify_index=67 value="\"\\xbb\\x05\\xf4\\xba\\x02\\n\\xb8\\x05\\n\\x0eprimary-grover\\x12\\xa5\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335362813Z caller=mock.go:196 msg=Get key=collectors/distributor modify_index=68 value="\"D\\xf0C\\nB\\n\\x0eprimary-grover\\x120\\n\\v127.0.0.1:0\\x10\\xfd\\xd5º\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335394724Z caller=mock.go:123 msg=CAS key=collectors/distributor modify_index=68 value="\"D\\xf0C\\nB\\n\\x0eprimary-grover\\x120\\n\\v127.0.0.1:0\\x10\\x8c\\xd6º\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335430094Z caller=mock.go:196 msg=Get key=collectors/pattern-ring modify_index=74 value="\"\\xbb\\x05\\xf4\\xba\\x02\\n\\xb8\\x05\\n\\x0eprimary-grover\\x12\\xa5\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335453275Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335460005Z caller=mock.go:196 msg=Get key=collectors/distributor modify_index=75 value="\"D\\xf0C\\nB\\n\\x0eprimary-grover\\x120\\n\\v127.0.0.1:0\\x10\\x8c\\xd6º\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.335514266Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.36283431Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=0
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.362853851Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=71 value="\"\\xbe\\x05\\xf4\\xbd\\x02\\n\\xbb\\x05\\n\\x0eprimary-grover\\x12\\xa8\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.362881871Z caller=mock.go:123 msg=CAS key=collectors/ring modify_index=71 value="\"\\xbe\\x05\\xf4\\xbd\\x02\\n\\xbb\\x05\\n\\x0eprimary-grover\\x12\\xa8\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.362904312Z caller=mock.go:196 msg=Get key=collectors/ring modify_index=76 value="\"\\xbe\\x05\\xf4\\xbd\\x02\\n\\xbb\\x05\\n\\x0eprimary-grover\\x12\\xa8\\x05\\n\\x0e127.0.0.1:9\""
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.362943182Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:36 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:36.83878007Z caller=push.go:162 org_id=fake traceID=00bef1550be02048 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.7 kB" streams=1 entries=27 streamLabelsSize="60 B" entriesSize="5.9 kB" structuredMetadataSize="0 B" totalSize="5.9 kB" mostRecentLagMs=272
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.335294261Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.411951541Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.411976301Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.411987931Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412011462Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412039822Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412050853Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412057653Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412062873Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412072583Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:37 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:37.412080263Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.138437219Z caller=push.go:162 org_id=fake traceID=1aef50287252f21b msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.0 kB" streams=1 entries=12 streamLabelsSize="60 B" entriesSize="2.2 kB" structuredMetadataSize="0 B" totalSize="2.3 kB" mostRecentLagMs=569
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.334700453Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412636229Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.41267085Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.41269458Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.41270275Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412708631Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412714701Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412720361Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412731921Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412737091Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:38 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:38.412741781Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.338429568Z caller=push.go:162 org_id=fake traceID=5c622fa0c1b01b6d msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="992 B" streams=1 entries=12 streamLabelsSize="60 B" entriesSize="2.2 kB" structuredMetadataSize="0 B" totalSize="2.3 kB" mostRecentLagMs=766
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413715698Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413738469Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413758909Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413765679Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413771079Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.41379903Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.41381206Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.41382148Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413832631Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:39 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:39.413839231Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.335260571Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414328202Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414346623Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414355073Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414370733Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414376313Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414380983Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414386553Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414391733Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414396914Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.414400514Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:40 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:40.638697711Z caller=push.go:162 org_id=fake traceID=0d6d1d8620d3be86 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.3 kB" streams=1 entries=22 streamLabelsSize="60 B" entriesSize="3.8 kB" structuredMetadataSize="0 B" totalSize="3.8 kB" mostRecentLagMs=63
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.334871809Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.41436631Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.41438236Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414389461Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414394831Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414399361Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414403531Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414418141Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.414422661Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:41 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:41.838802627Z caller=push.go:162 org_id=fake traceID=1dfc9685a42dda32 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="926 B" streams=1 entries=10 streamLabelsSize="60 B" entriesSize="1.9 kB" structuredMetadataSize="0 B" totalSize="1.9 kB" mostRecentLagMs=261
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.095502087Z caller=flush.go:214 msg="computed flush rate" rate=3.3333333333333335
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.334875208Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414876798Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414901599Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414911929Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414918629Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414924669Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414929839Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.414948241Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.415032332Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.415056103Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:42 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:42.415067263Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.138241373Z caller=push.go:162 org_id=fake traceID=4b9f094bb8a01128 msg="push request parsed" path=/loki/api/v1/push contentType=application/x-protobuf contentEncoding= bodySize="1.1 kB" streams=1 entries=13 streamLabelsSize="60 B" entriesSize="2.4 kB" structuredMetadataSize="0 B" totalSize="2.4 kB" mostRecentLagMs=558
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.334521722Z caller=tee_service.go:257 component=pattern-tee msg="prepared pattern Tee batches for tenant" tenant=fake stream_count=1
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415667909Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/compactor
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415694619Z caller=mock.go:150 msg=Get key=collectors/compactor wait_index=73
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415701279Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/distributor
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.41572383Z caller=mock.go:150 msg=Get key=collectors/distributor wait_index=75
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.4157339Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/pattern-ring
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.41574996Z caller=mock.go:150 msg=Get key=collectors/pattern-ring wait_index=74
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415761181Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/ring
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415776371Z caller=mock.go:150 msg=Get key=collectors/ring wait_index=76
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415790831Z caller=mock.go:186 msg="Get - deadline exceeded" key=collectors/scheduler
Dec 4 14:18:43 primary-grover loki[51934]: level=debug ts=2024-12-04T19:18:43.415796791Z caller=mock.go:150 msg=Get key=collectors/scheduler wait_index=72
Should I perhaps make any modifications to promtail yaml, and try and limit the data being sent out?
I still think it’s some sort of resource contention.
Are you running Loki in a container or baremetal installation? If you are running in container, double check and make sure you are giving it adequate amount of memory and CPU. Make sure file handler limit is set sufficiently high as well. I would also recommend checking all system metrics and see if you spot any bottleneck somewhere.
Hi. Sorry for the delayed response. Loki is running on a VM, albeit a dedicated instance. I will continue to poke around but i am just not seeing anything that strokes me as odd … yet