Error processing requests-rpc error: code = Canceled desc = context canceled

Grafana Loki
1

On Monolithic setup, Loki 2.9.14 + S3 + memberlist

gets the below error while searching in Grafana. Please find attached the configuration also.

level=error ts=2025-06-11T12:47:26.874670158Z caller=frontend_processor.go:69 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
level=error ts=2025-06-11T12:47:26.87498672Z caller=frontend_processor.go:145 msg="error processing requests" err=EOF

Loki configuration:

# Loki 2.9.14 + S3 + memberlist - prod

# target:

auth_enabled: false

ballast_bytes: 10000  # default = 0

server:
  # http_listen_address: 127.0.0.1  # default = "" - comment in production
  http_listen_port: 3101  # 3101 - make in production
  grpc_listen_address: 127.0.0.1  # default = ""
  grpc_listen_port: 9095
  http_server_read_timeout: 10m  # default = 30s
  http_server_write_timeout: 10m  # default = 30s
  http_server_idle_timeout: 5m  # default = 2m
  grpc_server_max_recv_msg_size: 41943040  # 40MB to handle long log lines, default = 4194304
  grpc_server_max_send_msg_size: 41943040  # 40MB to handle long log lines, default = 4194304
  grpc_server_max_concurrent_streams: 0 # default = 100 (0 = unlimited)
  log_level: warn

distributor:
  ring:
    kvstore:
      store: memberlist
    heartbeat_period: 0 # default = 5s
    heartbeat_timeout: 0 # default = 1m

querier:
  tail_max_duration: 1h  # default = 1h
  query_ingesters_within: 0 # default = 3h
  engine:
    timeout: 5m # default = 5m
    max_look_back_period: 168h  # 7 days, default = 30s
  max_concurrent: 4  # default = 4, 2-core CPU → 4, 4-core CPU → 8

query_scheduler:
  max_outstanding_requests_per_tenant: 64000 # default = 32000
  grpc_client_config:
    max_recv_msg_size: 209715200 # default = 104857600 (bytes)
    max_send_msg_size: 209715200 # default = 104857600 (bytes)
    grpc_compression: 'snappy'  # default = ""
    backoff_on_ratelimits: true # default = false
    connect_timeout: 0s # default = 5s
  use_scheduler_ring: false  # default = false

frontend:
  log_queries_longer_than: 60s  # default = 0s
  max_outstanding_per_tenant: 8192  # default = 2048
  grpc_client_config:
    max_recv_msg_size: 209715200 # default = 104857600 (bytes)
    max_send_msg_size: 209715200 # default = 104857600 (bytes)
    grpc_compression: 'snappy'  # default = ""
    backoff_on_ratelimits: true # default = false
    connect_timeout: 0s # default = 5s
  compress_responses: true

query_range:
  align_queries_with_step: true # default = false
  results_cache:
    cache:
      embedded_cache:
        enabled: true # default = false
    compression: 'snappy'  # default = ""
  cache_results: true # default = false
  max_retries: 15  # default = 5
  index_stats_results_cache:
    cache:
      embedded_cache:
        enabled: true # default = false
    compression: 'snappy'  # default = ""

ruler:
  storage:
    type: local
    local:
      directory: /data/loki/rules
  rule_path: /data/loki/rules/fake/
  alertmanager_url: http://127.0.0.1:9093
  enable_api: true

ingester_client:
  pool_config:
    client_cleanup_period: 20s  # default = 15s
    health_check_ingesters: false # default = true
    remote_timeout: 5s  # default = 1s
  remote_timeout: 10s # default = 5s
  grpc_client_config:
    max_recv_msg_size: 209715200 # default = 104857600 (bytes)
    max_send_msg_size: 209715200 # default = 104857600 (bytes)
    grpc_compression: 'snappy'  # default = ""
    backoff_on_ratelimits: true # default = false
    connect_timeout: 0s # default = 5s

ingester:
  concurrent_flushes: 64  # default = 32
  flush_check_period: 30s # default = 30s
  flush_op_timeout: 10m # default = 10m
  chunk_retain_period: 5s  # To ease recovery, default = 0s
  chunk_idle_period: 30m  # Close chunks after no more writes, default = 30m
  chunk_block_size: 262144  # 256KB, default = 262144
  chunk_target_size: 1572864  # ~1.5MB target chunk size, default = 1572864
  chunk_encoding: "snappy"  # default = "gzip"
  max_chunk_age: 1h # default = 2h
  autoforget_unhealthy: false # default = false
  query_store_max_look_back_period: 0s  # default = 0s
  wal:
    enabled: true # default = true
    dir: /data/loki/wal

index_gateway:
  mode: "simple"  # default = "simple"

storage_config:
  tsdb_shipper:
    active_index_directory: /data/loki/tsdb-index
    shared_store: s3
    shared_store_key_prefix: "index/" # default = "index/"
    cache_location: /data/loki/tsdb-cache
    cache_ttl: 24h  # Can be increased for faster performance over longer query periods, uses more disk space
    resync_interval: 10m  # default = 5m
    query_ready_num_days: 1 # default = 0
    index_gateway_client:
      grpc_client_config:
        max_recv_msg_size: 209715200 # default = 104857600 (bytes)
        max_send_msg_size: 209715200 # default = 104857600 (bytes)
        grpc_compression: 'snappy'  # default = ""
        backoff_on_ratelimits: true # default = false
        connect_timeout: 0s # default = 5s
      server_address: 127.0.0.1 # default = ""
  filesystem:
    directory: /data/loki/chunks
  aws:
    # s3: s3://<bucketName>
    s3forcepathstyle: true  # default = false
    bucketnames: <bucketName> # default = ""
    endpoint: s3.me-central-1.amazonaws.com # default = ""
    region: me-central-1  # default = ""
    # access_key_id: ${AWS_ACCESS_KEY_ID} # default = ""
    # secret_access_key: ${AWS_SECRET_ACCESS_KEY} # default = ""
    # session_token: ${AWS_SESSION_TOKEN} # default = ""
    insecure: false # default = false
    sse_encryption: false # default = false
  index_queries_cache_config:
    embedded_cache:
      enabled: true # default = false
  max_parallel_get_chunk: 150 # default = 150
  max_chunk_batch_size: 50 # default = 50

chunk_store_config:
  chunk_cache_config:
    embedded_cache:
      enabled: true # default = false

schema_config:
  configs:
  - from: 2022-01-01
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
    chunks:
      prefix: chunks_
      period: 24h

compactor:
  working_directory: /data/loki/compactor
  compaction_interval: 5m # default = 10m
  retention_enabled: true # default = false
  retention_delete_delay: 1h  # default = 2h
  retention_delete_worker_count: 150  # default = 150
  delete_request_store: s3  # default = ""
  delete_batch_size: 100  # default = 70
  max_compaction_parallelism: 2 # default = 1
  tables_to_compact: 0  # default = 0

limits_config:
  ingestion_rate_mb: 500  # 500MB/s, default = 4
  ingestion_burst_size_mb: 1000  # Burst capability for spikes, default = 6
  max_label_name_length: 1024 # default = 1024
  max_label_value_length: 5120  # default = 2048
  reject_old_samples: true
  reject_old_samples_max_age: 2w  # default = 1w (window) = max_chunk_age/2
  max_line_size: 0B # 10MB to handle long log lines, default = 0B
  max_global_streams_per_user: 100000 # default = 5000
  per_stream_rate_limit: 20MB # default = 3MB
  per_stream_rate_limit_burst: 40MB # default = 15MB
  max_query_parallelism: 64 # default = 32
  tsdb_max_query_parallelism: 32  # default = 128
  max_entries_limit_per_query: 5000  # default = 5000
  split_queries_by_interval: 1h # default = 1h
  volume_enabled: true
  retention_period: 168h  # 7 days retention

frontend_worker:
  frontend_address: 127.0.0.1:9095 # default = ""
  # scheduler_address: 127.0.0.1:9095 # default = ""
  grpc_client_config:
    max_recv_msg_size: 209715200 # default = 104857600 (bytes)
    max_send_msg_size: 209715200 # default = 104857600 (bytes)
    grpc_compression: 'snappy'  # default = ""
    backoff_on_ratelimits: true # default = false
    connect_timeout: 0s # default = 5s

table_manager:
  retention_deletes_enabled: true # default = false
  retention_period: 168h  # 7 days retention

memberlist:
 gossip_interval: 200ms  # default = 200ms
 gossip_nodes: 1  # default = 3
 gossip_to_dead_nodes_time: 30s # default = 30s
 advertise_addr: 127.0.0.1 # default = ""
 advertise_port: 7946 # default = 7946
 join_members: # default = []
   - 127.0.0.1:7946
 left_ingesters_timeout: 5m # default = 5m
 bind_addr: ['127.0.0.1']  # default = []
 bind_port: 7946 # default = 7946

# runtime_config:

# tracing:

analytics:
  reporting_enabled: false

common:
  path_prefix: /data/loki
  storage:
    s3:
      s3forcepathstyle: true  # default = false
      bucketnames: <bucketName> # default = ""
      endpoint: s3.me-central-1.amazonaws.com # default = ""
      region: me-central-1  # default = ""
      # access_key_id: ${AWS_ACCESS_KEY_ID} # default = ""
      # secret_access_key: ${AWS_SECRET_ACCESS_KEY} # default = ""
      # session_token: ${AWS_SESSION_TOKEN} # default = ""
      insecure: false # default = false
      sse_encryption: false # default = false
  persist_tokens: true
  replication_factor: 1
  ring:
    kvstore:
      store: memberlist
      prefix: "collectors/" # default = "collectors/"
    heartbeat_period: 0 # default = 15s
    heartbeat_timeout: 0  # default = 1m
    instance_port: 9095 # default = 0
    instance_addr: 127.0.0.1
  instance_addr: 127.0.0.1
  compactor_address: 127.0.0.1:3100  # default = ""
  compactor_grpc_address: 127.0.0.1:9095  # default = ""

shutdown_delay: 3s  # # default = 0s
2
  1. What does your loki container resource usage look like?
  2. Do you see any other error message before the RCP error?
3

  1. What does your Loki container resource usage look like?
    Ans: - since the last 1 week or the period when the error occurred (Loki on EC2)

    1. CPU % utilization - 4-5 % range
    2. mem_used_percent - 12-13 % range
    3. disk_used_percent - 14-15 % range
    4. mem_free - 400-500 M range

  2. Do you see any other error message before the RCP error?
    Ans: Yes, below was error just before rpc error

level=error ts=2025-06-11T12:45:21.591758767Z caller=http.go:320 org_id=fake traceID=1858c939ae8a4460 msg="Unexpected error from client" err="read tcp 10.41.3.89:3101->172.20.12.5:63163: read: connection reset by peer"
level=error ts=2025-06-11T12:45:43.61537618Z caller=http.go:320 org_id=fake traceID=68d1ad137b4a9b6f msg="Unexpected error from client" err="read tcp 10.41.3.89:3101->172.20.12.5:63166: read: connection reset by peer"
level=error ts=2025-06-11T12:46:21.06735062Z caller=http.go:320 org_id=fake traceID=4ed0bec6715445f6 msg="Unexpected error from client" err="read tcp 10.41.3.89:3101->172.20.12.5:63183: read: connection reset by peer"
level=error ts=2025-06-11T12:46:37.579577091Z caller=frontend_processor.go:69 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
level=error ts=2025-06-11T12:46:37.579632886Z caller=retry.go:73 org_id=fake traceID=2decf77901c9995e msg="error processing request" try=0 query="{name=\"loki-canary\"} |= \"\"" err="context canceled"
level=error ts=2025-06-11T12:47:25.259761856Z caller=frontend_processor.go:69 msg="error processing requests" address=127.0.0.1:9095 err="rpc error: code = Canceled desc = context canceled"
level=error ts=2025-06-11T12:47:25.259833429Z caller=retry.go:73 org_id=fake traceID=70584f41e20bf3e3 msg="error processing request" try=0 query="{stream=\"stdout\"} |= \"\"" err="context canceled"
4
  1. Does it work some of the time or does it not work at all?
  2. I see you are using 127.0.0.1 for most of your endpoints, is there a reason for this?
5
  1. Does it work some of the time, or does it not work at all?
    Ans: Yes, it does work sometimes and sometimes not.
  2. I see you are using 127.0.0.1 for most of your endpoints, is there a reason for this?
    Ans: Yes, I am having a monolithic setup on a single VM/node locally, that is why I am using 127.0.0.1.

Anyway, do you find anything wrong/mistake in the configuration for this Loki v2.9.14?

6

I don’t see anything obviously wrong. If it’s context error I’d check resource usage and see if there is any error message from Loki on some sort of limits being hit.

I would also recommend you to change to 0.0.0.0 just as a test and see if that helps.