Loki + s3 + grafana alert history

Grafana Loki
, ,
1

Currently we have deployed Loki with Grafana and configured alert history correctly. However, in the s3 bucket where chunks and indexes are stored, recently there are no indexes inside the index folder and there is a delete_requests.gz file in the following path:

https://s3.my-region.amazonaws.com/my-bucket-for-loki/index/delete_requests/delete_requests.gz

In the logs we found the following:

caller=util.go:94 file-name=delete_requests.gz size="16 kB" msg="downloaded file"
caller=manager.go:122 msg="deleted rule manager and local rule files" user=fake

We did some tests with the ruler (first local, then s3 storage, local again with another loki instance to work as ruler and then s3 storage definitely with a monolithic deployment), so we suspect this might be related. Is this the expected behaviour (to empty the index folder in the s3 and leave that file)? Note also that we have two different buckets: one for the ruler and one for the index so this shouldn’t have happened, right? Strangely enough, we are able to query our logs just fine.

Additionally, in order to enable alert history Grafana required Loki ruler to have a s3 storage. Why? We have checked the s3 bucket and no rule file is created once we have added the alert rules via Grafana UI.

Just in case this is useful, here is our ruler config:

ruler:
  storage:
    type: s3
    s3:
      s3: s3://my-region/my-loki-ruler-bucket
  rule_path: /var/tmp/rules
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_v2: true
  ring:
    kvstore:
      store: inmemory
      prefix: rulers/
  enable_api: true

Thanks in advance!

2

Your issues seem to be all over the place. I’ll try to break it down a bit:

  1. The delete_request.gz file is from compactor if i remember correctly, not from ruler. And unless you specifically set it to write to your S3 backend it should be stored locally (unless this changed with recent release).

  2. If you are able to query your logs there is no way all your index and chunk files are gone. I’d double check your configuration and make sure you understand where your data actually are.

  3. I don’t use alert history, so I can’t comment on it. But in general Grafana requires Loki ruler to have S3 storage if you wish to interact with ruler at all from Grafana. Because in order for Grafana to interact with ruler, ruler needs to have API enabled, and in order to do that ruler needs to have S3 storage as backend. Without S3 backend ruler API is not available, therefore Grafana can’t interact with ruler without it.

3

Tony, thanks for your answer. May be the way I explained myself wasn’t clear. Currently we have two questions:

  1. why does the compactor delete the indexes in the s3 even when the limits_config is set to 0s and there is no retention_enabled? We managed to figure out it was the compactor deleting the indexes because we removed the deleteObject permissions and began seeing an error of unauthorized in the logs.

  2. why does grafana need loki to have an s3 storage for the ruler, when it doesn’t interact with it? (Or at least, doesn’t seem so because no rules files are being created inside the s3 bucket)

Regarding point 2 (querying chunks with indexes gone) we do have a suspicion: when we changed from file system to s3 to filesystem again, some of the indexes were kept in the filesystem.

I would really appreciate your comments, thanks! :slight_smile:

4

  1. Can you share your Loki configuration, please? Just to be safe, print it out from /config endpoint if you can.

  2. If you are adding Loki to Grafana’s alert view, then it is interacting with Loki ruler. Otherwise if you are sure you don’t need to see Loki ruler from Grafana then you can use filesystem storage and not have API for ruler.

5
  1. Sure, here is our Loki config:
auth_enabled: false

server:
  http_listen_port: 3100
  http_server_read_timeout: 360s
  http_server_write_timeout: 360s
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608

ingester:
  chunk_encoding: snappy
  wal:
    enabled: true
    dir: /loki/wal

common:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /loki

schema_config:
  configs:
    - from: "2020-05-15"
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

analytics:
  reporting_enabled: false

query_scheduler:
  max_outstanding_requests_per_tenant: 10000

compactor:
  shared_store: s3
  working_directory: /loki/compactor

storage_config:
  aws:
    region: my-region
    bucketnames: my-bucket-name
    s3: s3://my-region/my-bucket-name

  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3

limits_config:
  query_timeout: 350s
  retention_period: 2160h
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  max_cache_freshness_per_query: 10m
  split_queries_by_interval: 24h
  per_stream_rate_limit: "5MB"
  per_stream_rate_limit_burst: "15MB"
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 100
  max_entries_limit_per_query: 1000
  max_label_value_length: 20480
  max_label_name_length: 10240
  max_label_names_per_series: 300


ruler:
  storage:
    type: s3
    s3:
      s3: s3://my-region/my-bucket-name-for-ruler
  rule_path: /var/tmp/rules
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_v2: true
  ring:
    kvstore:
      store: inmemory
      prefix: rulers/
  enable_api: true

Configured as it is, IT IS trying to delete our indexes; at least this is what we see in the logs:


level=info ts=2024-05-29T22:36:29.398936862Z caller=index_set.go:269 table-name=index_19872 user-id=fake user-id=fake msg="removing source db files from storage" count=126

We tried granting deleteObject to the role, and the indexes were marked for deletion. We recovered them with this script provided by aws Retrieve an Amazon S3 object that was deleted | AWS re:Post and removed the deleteObject permission to the role; until we understand how this works.

  1. Regarding this, we configured the ruler withe filesystem storage, but got the following error:

level=error msg="GetRuleGroup unsupported in rule local store\n" error="GetRuleGroup unsupported in rule local store\n"

I understand the error comes from here: grafana/public/app/features/alerting/unified/api/buildInfo.ts at 98152b2f84b685d25ae3471a24e6630eedf00ba0 · grafana/grafana · GitHub

Due to this error we configured Loki ruler to use s3 as storage, and the error went away. However, after creating the rules in Grafana (in order to use alert history, alerts must be managed by Grafana) we kind of expected Grafana to save the rules in the s3 bucket of the rule. Plot twist, it doesn’t. So we don’t see the point of Grafana requiring Loki ruler storage to be non-local.

And thanks again for your time :sunny:

6

In the config shared above the limits_config retention_period is set to 2160h because we’ve grown suspicious of the 0s default.

7

  1. I am suspicious of your Loki configuration being what you actually set it to be. According to documentation compactor.retention_enabled is default to false, meaning by not setting any retention related configurations it would not have any retention set at all. So the fact that you are not setting it should mean that logs are stored forever. I’d encourage you to print a list of the entire configuration from the /config endpoint and double check.

  2. So you are indeed interacting with ruler from Grafana. I would also hazard a guess that whatever rules you created in Grafana are Grafana-stored rules / alerts, and are not actually stored on Loki ruler. You can confirm this by actually querying the Loki ruler from an API call.

8

Here is the full config from /config:

target: all
http_prefix: ""
ballast_bytes: 0
use_buffered_logger: true
use_sync_logger: true
server:
  http_listen_network: tcp
  http_listen_address: ""
  http_listen_port: 3100
  http_listen_conn_limit: 0
  grpc_listen_network: tcp
  grpc_listen_address: ""
  grpc_listen_port: 9095
  grpc_listen_conn_limit: 0
  tls_cipher_suites: ""
  tls_min_version: ""
  http_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: true
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 6m0s
  http_server_write_timeout: 6m0s
  http_server_idle_timeout: 2m0s
  grpc_server_max_recv_msg_size: 8388608
  grpc_server_max_send_msg_size: 8388608
  grpc_server_max_concurrent_streams: 100
  grpc_server_max_connection_idle: 2562047h47m16.854775807s
  grpc_server_max_connection_age: 2562047h47m16.854775807s
  grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
  grpc_server_keepalive_time: 2h0m0s
  grpc_server_keepalive_timeout: 20s
  grpc_server_min_time_between_pings: 10s
  grpc_server_ping_without_stream_allowed: true
  log_format: logfmt
  log_level: info
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  log_request_headers: false
  log_request_at_info_level_enabled: false
  log_request_exclude_headers_list: ""
  http_path_prefix: ""
internal_server:
  http_listen_network: tcp
  http_listen_address: localhost
  http_listen_port: 3101
  http_listen_conn_limit: 0
  grpc_listen_network: ""
  grpc_listen_address: ""
  grpc_listen_port: 0
  grpc_listen_conn_limit: 0
  tls_cipher_suites: ""
  tls_min_version: ""
  http_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  grpc_tls_config:
    cert_file: ""
    key_file: ""
    client_auth_type: ""
    client_ca_file: ""
  register_instrumentation: false
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 30s
  http_server_write_timeout: 30s
  http_server_idle_timeout: 2m0s
  grpc_server_max_recv_msg_size: 0
  grpc_server_max_send_msg_size: 0
  grpc_server_max_concurrent_streams: 0
  grpc_server_max_connection_idle: 0s
  grpc_server_max_connection_age: 0s
  grpc_server_max_connection_age_grace: 0s
  grpc_server_keepalive_time: 0s
  grpc_server_keepalive_timeout: 0s
  grpc_server_min_time_between_pings: 0s
  grpc_server_ping_without_stream_allowed: false
  log_format: ""
  log_level: ""
  log_source_ips_enabled: false
  log_source_ips_header: ""
  log_source_ips_regex: ""
  log_request_headers: false
  log_request_at_info_level_enabled: false
  log_request_exclude_headers_list: ""
  http_path_prefix: ""
  enable: false
distributor:
  ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_enable_ipv6: false
  rate_store:
    max_request_parallelism: 200
    stream_rate_update_interval: 1s
    ingester_request_timeout: 500ms
    debug: false
  write_failures_logging:
    rate: 1KB
    add_insights_label: false
querier:
  tail_max_duration: 1h0m0s
  query_ingesters_within: 3h0m0s
  engine:
    timeout: 5m0s
    max_look_back_period: 30s
  max_concurrent: 10
  query_store_only: false
  query_ingester_only: false
  multi_tenant_queries_enabled: false
  query_timeout: 0s
  per_request_limits_enabled: false
query_scheduler:
  max_outstanding_requests_per_tenant: 10000
  max_queue_hierarchy_levels: 3
  querier_forget_delay: 0s
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: ""
    rate_limit: 0
    rate_limit_burst: 0
    backoff_on_ratelimits: false
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10
    initial_stream_window_size: 63KiB1023B
    initial_connection_window_size: 63KiB1023B
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    connect_timeout: 5s
    connect_backoff_base_delay: 1s
    connect_backoff_max_delay: 5s
  use_scheduler_ring: true
  scheduler_ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    tokens_file_path: ""
    zone_awareness_enabled: false
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_availability_zone: ""
    instance_enable_ipv6: false
frontend:
  log_queries_longer_than: 0s
  max_body_size: 10485760
  query_stats_enabled: false
  max_outstanding_per_tenant: 2048
  querier_forget_delay: 0s
  scheduler_address: ""
  scheduler_dns_lookup_period: 10s
  scheduler_worker_concurrency: 5
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: ""
    rate_limit: 0
    rate_limit_burst: 0
    backoff_on_ratelimits: false
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10
    initial_stream_window_size: 63KiB1023B
    initial_connection_window_size: 63KiB1023B
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    connect_timeout: 5s
    connect_backoff_base_delay: 1s
    connect_backoff_max_delay: 5s
  graceful_shutdown_timeout: 5m0s
  instance_interface_names:
  - eth0
  - lo
  address: ""
  port: 0
  compress_responses: false
  downstream_url: ""
  tail_proxy_url: ""
  tail_tls_config:
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
query_range:
  split_queries_by_interval: 0s
  align_queries_with_step: false
  results_cache:
    cache:
      enable_fifocache: true
      default_validity: 1h0m0s
      background:
        writeback_goroutines: 10
        writeback_buffer: 10000
        writeback_size_limit: 1GB
      memcached:
        expiration: 0s
        batch_size: 1024
        parallelism: 100
      memcached_client:
        host: ""
        service: memcached
        addresses: ""
        timeout: 100ms
        max_idle_conns: 16
        max_item_size: 0
        update_interval: 1m0s
        consistent_hash: true
        circuit_breaker_consecutive_failures: 10
        circuit_breaker_timeout: 10s
        circuit_breaker_interval: 10s
      redis:
        endpoint: ""
        master_name: ""
        timeout: 500ms
        expiration: 0s
        db: 0
        pool_size: 0
        username: ""
        password: ""
        tls_enabled: false
        tls_insecure_skip_verify: false
        idle_timeout: 0s
        max_connection_age: 0s
        route_randomly: false
      embedded_cache:
        max_size_mb: 100
        ttl: 1h0m0s
      fifocache:
        max_size_bytes: 1GB
        max_size_items: 0
        ttl: 1h0m0s
        validity: 0s
        size: 0
        purgeinterval: 0s
      prefix: frontend.
      async_cache_write_back_concurrency: 16
      async_cache_write_back_buffer_size: 500
    compression: ""
  cache_results: false
  max_retries: 5
  parallelise_shardable_queries: true
  forward_headers_list: []
  required_query_response_format: json
  cache_index_stats_results: false
  index_stats_results_cache:
    cache:
      enable_fifocache: true
      default_validity: 1h0m0s
      background:
        writeback_goroutines: 10
        writeback_buffer: 10000
        writeback_size_limit: 1GB
      memcached:
        expiration: 0s
        batch_size: 1024
        parallelism: 100
      memcached_client:
        host: ""
        service: memcached
        addresses: ""
        timeout: 100ms
        max_idle_conns: 16
        max_item_size: 0
        update_interval: 1m0s
        consistent_hash: true
        circuit_breaker_consecutive_failures: 10
        circuit_breaker_timeout: 10s
        circuit_breaker_interval: 10s
      redis:
        endpoint: ""
        master_name: ""
        timeout: 500ms
        expiration: 0s
        db: 0
        pool_size: 0
        username: ""
        password: ""
        tls_enabled: false
        tls_insecure_skip_verify: false
        idle_timeout: 0s
        max_connection_age: 0s
        route_randomly: false
      embedded_cache:
        max_size_mb: 100
        ttl: 1h0m0s
      fifocache:
        max_size_bytes: 1GB
        max_size_items: 0
        ttl: 1h0m0s
        validity: 0s
        size: 0
        purgeinterval: 0s
      prefix: frontend.
      async_cache_write_back_concurrency: 16
      async_cache_write_back_buffer_size: 500
    compression: ""
ruler:
  external_url: ""
  datasource_uid: ""
  ruler_client:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: ""
    rate_limit: 0
    rate_limit_burst: 0
    backoff_on_ratelimits: false
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10
    initial_stream_window_size: 63KiB1023B
    initial_connection_window_size: 63KiB1023B
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    connect_timeout: 5s
    connect_backoff_base_delay: 1s
    connect_backoff_max_delay: 5s
  evaluation_interval: 1m0s
  poll_interval: 1m0s
  storage:
    type: s3
    azure:
      environment: AzureGlobal
      account_name: ""
      account_key: ""
      container_name: loki
      endpoint_suffix: ""
      use_managed_identity: false
      use_federated_token: false
      user_assigned_id: ""
      use_service_principal: false
      client_id: ""
      client_secret: ""
      tenant_id: ""
      chunk_delimiter: '-'
      download_buffer_size: 512000
      upload_buffer_size: 256000
      upload_buffer_count: 1
      request_timeout: 30s
      max_retries: 5
      min_retry_delay: 10ms
      max_retry_delay: 500ms
    alibabacloud:
      bucket: ""
      endpoint: ""
      access_key_id: ""
      secret_access_key: ""
    gcs:
      bucket_name: ""
      service_account: ""
      chunk_buffer_size: 0
      request_timeout: 0s
      enable_opencensus: true
      enable_http2: true
    s3:
      s3: s3://my-region/my-buclet.loki-rules
      s3forcepathstyle: false
      bucketnames: ""
      endpoint: ""
      region: ""
      access_key_id: ""
      secret_access_key: ""
      session_token: ""
      insecure: false
      sse_encryption: false
      http_config:
        timeout: 0s
        idle_conn_timeout: 1m30s
        response_header_timeout: 0s
        insecure_skip_verify: false
        ca_file: ""
      signature_version: v4
      storage_class: STANDARD
      sse:
        type: ""
        kms_key_id: ""
        kms_encryption_context: ""
      backoff_config:
        min_period: 100ms
        max_period: 3s
        max_retries: 5
    bos:
      bucket_name: ""
      endpoint: bj.bcebos.com
      access_key_id: ""
      secret_access_key: ""
    swift:
      auth_version: 0
      auth_url: ""
      internal: false
      username: ""
      user_domain_name: ""
      user_domain_id: ""
      user_id: ""
      password: ""
      domain_id: ""
      domain_name: ""
      project_id: ""
      project_name: ""
      project_domain_id: ""
      project_domain_name: ""
      region_name: ""
      container_name: ""
      max_retries: 3
      connect_timeout: 10s
      request_timeout: 5s
    cos:
      forcepathstyle: false
      bucketnames: ""
      endpoint: ""
      region: ""
      access_key_id: ""
      secret_access_key: ""
      http_config:
        idle_conn_timeout: 1m30s
        response_header_timeout: 0s
      backoff_config:
        min_period: 100ms
        max_period: 3s
        max_retries: 5
      api_key: ""
      service_instance_id: ""
      auth_endpoint: https://iam.cloud.ibm.com/identity/token
      cr_token_file_path: ""
      trusted_profile_name: ""
      trusted_profile_id: ""
    local:
      directory: ""
  rule_path: /var/tmp/rules
  alertmanager_url: http://alertmanager:9093
  enable_alertmanager_discovery: false
  alertmanager_refresh_interval: 1m0s
  enable_alertmanager_v2: true
  notification_queue_capacity: 10000
  notification_timeout: 10s
  alertmanager_client:
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    basic_auth_username: ""
    basic_auth_password: ""
    type: Bearer
  for_outage_tolerance: 1h0m0s
  for_grace_period: 10m0s
  resend_delay: 1m0s
  enable_sharding: false
  sharding_strategy: default
  sharding_algo: by-group
  search_pending_for: 5m0s
  ring:
    kvstore:
      store: inmemory
      prefix: rulers/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_enable_ipv6: false
    num_tokens: 128
  flush_period: 1m0s
  enable_api: true
  enabled_tenants: ""
  disabled_tenants: ""
  query_stats_enabled: false
  disable_rule_group_label: false
  wal:
    tenant: ""
    name: ""
    remotewrite: []
    dir: ruler-wal
    truncate_frequency: 1h0m0s
    min_age: 5m0s
    max_age: 4h0m0s
  wal_cleaner:
    min_age: 12h0m0s
    deprecatedperiod: 0s
  remote_write:
    enabled: false
    config_refresh_period: 10s
  evaluation:
    mode: local
    max_jitter: 0s
ingester_client:
  pool_config:
    client_cleanup_period: 15s
    health_check_ingesters: true
    remote_timeout: 1s
  remote_timeout: 5s
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: ""
    rate_limit: 0
    rate_limit_burst: 0
    backoff_on_ratelimits: false
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10
    initial_stream_window_size: 63KiB1023B
    initial_connection_window_size: 63KiB1023B
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    connect_timeout: 5s
    connect_backoff_base_delay: 1s
    connect_backoff_max_delay: 5s
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
        prefix: collectors/
        consul:
          host: localhost:8500
          acl_token: ""
          http_client_timeout: 20s
          consistent_reads: false
          watch_rate_limit: 1
          watch_burst_size: 1
          cas_retry_delay: 1s
        etcd:
          endpoints: []
          dial_timeout: 10s
          max_retries: 10
          tls_enabled: false
          tls_cert_path: ""
          tls_key_path: ""
          tls_ca_path: ""
          tls_server_name: ""
          tls_insecure_skip_verify: false
          tls_cipher_suites: ""
          tls_min_version: ""
          username: ""
          password: ""
        multi:
          primary: ""
          secondary: ""
          mirror_enabled: false
          mirror_timeout: 2s
      heartbeat_timeout: 1m0s
      replication_factor: 1
      zone_awareness_enabled: false
      excluded_zones: ""
    num_tokens: 128
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    observe_period: 0s
    join_after: 0s
    min_ready_duration: 15s
    interface_names:
    - eth0
    - lo
    enable_inet6: false
    final_sleep: 0s
    tokens_file_path: ""
    availability_zone: ""
    unregister_on_shutdown: true
    readiness_check_ring_health: true
    address: 127.0.0.1
    port: 0
    id: 18b4ed97d8b9
  concurrent_flushes: 32
  flush_check_period: 30s
  flush_op_timeout: 10m0s
  chunk_retain_period: 0s
  chunk_idle_period: 30m0s
  chunk_block_size: 262144
  chunk_target_size: 1572864
  chunk_encoding: snappy
  max_chunk_age: 2h0m0s
  autoforget_unhealthy: false
  sync_period: 0s
  sync_min_utilization: 0
  max_returned_stream_errors: 10
  query_store_max_look_back_period: 2h41m0s
  wal:
    enabled: true
    dir: /loki/wal
    checkpoint_duration: 5m0s
    flush_on_shutdown: false
    replay_memory_ceiling: 4GB
  index_shards: 32
  max_dropped_streams: 10
  shutdown_marker_path: /loki
index_gateway:
  mode: simple
  ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    tokens_file_path: ""
    zone_awareness_enabled: false
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_availability_zone: ""
    instance_enable_ipv6: false
    replication_factor: 1
storage_config:
  alibabacloud:
    bucket: ""
    endpoint: ""
    access_key_id: ""
    secret_access_key: ""
  aws:
    dynamodb:
      dynamodb_url: ""
      api_limit: 2
      throttle_limit: 10
      metrics:
        url: ""
        target_queue_length: 100000
        scale_up_factor: 1.3
        ignore_throttle_below: 1
        queue_length_query: sum(avg_over_time(cortex_ingester_flush_queue_length{job="cortex/ingester"}[2m]))
        write_throttle_query: sum(rate(cortex_dynamo_throttled_total{operation="DynamoDB.BatchWriteItem"}[1m]))
          by (table) > 0
        write_usage_query: sum(rate(cortex_dynamo_consumed_capacity_total{operation="DynamoDB.BatchWriteItem"}[15m]))
          by (table) > 0
        read_usage_query: sum(rate(cortex_dynamo_consumed_capacity_total{operation="DynamoDB.QueryPages"}[1h]))
          by (table) > 0
        read_error_query: sum(increase(cortex_dynamo_failures_total{operation="DynamoDB.QueryPages",error="ProvisionedThroughputExceededException"}[1m]))
          by (table) > 0
      chunk_gang_size: 10
      chunk_get_max_parallelism: 32
      backoff_config:
        min_period: 100ms
        max_period: 50s
        max_retries: 20
      kms_key_id: ""
    s3: s3://my-region/my-buclet
    s3forcepathstyle: false
    bucketnames: my-buclet
    endpoint: ""
    region: my-region
    access_key_id: ""
    secret_access_key: ""
    session_token: ""
    insecure: false
    sse_encryption: false
    http_config:
      timeout: 0s
      idle_conn_timeout: 1m30s
      response_header_timeout: 0s
      insecure_skip_verify: false
      ca_file: ""
    signature_version: v4
    storage_class: STANDARD
    sse:
      type: ""
      kms_key_id: ""
      kms_encryption_context: ""
    backoff_config:
      min_period: 100ms
      max_period: 3s
      max_retries: 5
  azure:
    environment: AzureGlobal
    account_name: ""
    account_key: ""
    container_name: loki
    endpoint_suffix: ""
    use_managed_identity: false
    use_federated_token: false
    user_assigned_id: ""
    use_service_principal: false
    client_id: ""
    client_secret: ""
    tenant_id: ""
    chunk_delimiter: '-'
    download_buffer_size: 512000
    upload_buffer_size: 256000
    upload_buffer_count: 1
    request_timeout: 30s
    max_retries: 5
    min_retry_delay: 10ms
    max_retry_delay: 500ms
  bos:
    bucket_name: ""
    endpoint: bj.bcebos.com
    access_key_id: ""
    secret_access_key: ""
  bigtable:
    project: ""
    instance: ""
    grpc_client_config:
      max_recv_msg_size: 104857600
      max_send_msg_size: 104857600
      grpc_compression: ""
      rate_limit: 0
      rate_limit_burst: 0
      backoff_on_ratelimits: false
      backoff_config:
        min_period: 100ms
        max_period: 10s
        max_retries: 10
      initial_stream_window_size: 63KiB1023B
      initial_connection_window_size: 63KiB1023B
      tls_enabled: true
      tls_cert_path: ""
      tls_key_path: ""
      tls_ca_path: ""
      tls_server_name: ""
      tls_insecure_skip_verify: false
      tls_cipher_suites: ""
      tls_min_version: ""
      connect_timeout: 5s
      connect_backoff_base_delay: 1s
      connect_backoff_max_delay: 5s
    table_cache_enabled: true
    table_cache_expiration: 30m0s
  gcs:
    bucket_name: ""
    service_account: ""
    chunk_buffer_size: 0
    request_timeout: 0s
    enable_opencensus: true
    enable_http2: true
  cassandra:
    addresses: ""
    port: 9042
    keyspace: ""
    consistency: QUORUM
    replication_factor: 3
    disable_initial_host_lookup: false
    SSL: false
    host_verification: true
    host_selection_policy: round-robin
    CA_path: ""
    tls_cert_path: ""
    tls_key_path: ""
    auth: false
    username: ""
    password: ""
    password_file: ""
    custom_authenticators: []
    timeout: 2s
    connect_timeout: 5s
    reconnect_interval: 1s
    max_retries: 0
    retry_max_backoff: 10s
    retry_min_backoff: 100ms
    query_concurrency: 0
    num_connections: 2
    convict_hosts_on_failure: true
    table_options: ""
  boltdb:
    directory: ""
  filesystem:
    directory: ""
  swift:
    auth_version: 0
    auth_url: ""
    internal: false
    username: ""
    user_domain_name: ""
    user_domain_id: ""
    user_id: ""
    password: ""
    domain_id: ""
    domain_name: ""
    project_id: ""
    project_name: ""
    project_domain_id: ""
    project_domain_name: ""
    region_name: ""
    container_name: ""
    max_retries: 3
    connect_timeout: 10s
    request_timeout: 5s
  grpc_store: {}
  hedging:
    at: 0s
    up_to: 2
    max_per_second: 5
  named_stores:
    aws: {}
    azure: {}
    bos: {}
    filesystem: {}
    gcs: {}
    alibabacloud: {}
    swift: {}
    cos: {}
  cos:
    forcepathstyle: false
    bucketnames: ""
    endpoint: ""
    region: ""
    access_key_id: ""
    secret_access_key: ""
    http_config:
      idle_conn_timeout: 1m30s
      response_header_timeout: 0s
    backoff_config:
      min_period: 100ms
      max_period: 3s
      max_retries: 5
    api_key: ""
    service_instance_id: ""
    auth_endpoint: https://iam.cloud.ibm.com/identity/token
    cr_token_file_path: ""
    trusted_profile_name: ""
    trusted_profile_id: ""
  index_cache_validity: 5m0s
  index_queries_cache_config:
    enable_fifocache: false
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 10
      writeback_buffer: 10000
      writeback_size_limit: 1GB
    memcached:
      expiration: 0s
      batch_size: 1024
      parallelism: 100
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      ttl: 1h0m0s
    fifocache:
      max_size_bytes: 1GB
      max_size_items: 0
      ttl: 1h0m0s
      validity: 0s
      size: 0
      purgeinterval: 0s
    prefix: store.index-cache-read.
    async_cache_write_back_concurrency: 16
    async_cache_write_back_buffer_size: 500
  disable_broad_index_queries: false
  max_parallel_get_chunk: 150
  max_chunk_batch_size: 50
  boltdb_shipper:
    active_index_directory: ""
    shared_store: ""
    shared_store_key_prefix: index/
    cache_location: ""
    cache_ttl: 24h0m0s
    resync_interval: 5m0s
    query_ready_num_days: 0
    index_gateway_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
          min_period: 100ms
          max_period: 10s
          max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
      log_gateway_requests: false
    use_boltdb_shipper_as_backup: false
    ingestername: 18b4ed97d8b9
    mode: RW
    ingesterdbretainperiod: 20m0s
    build_per_tenant_index: false
  tsdb_shipper:
    active_index_directory: /loki/index
    shared_store: s3
    shared_store_key_prefix: index/
    cache_location: /loki/index_cache
    cache_ttl: 24h0m0s
    resync_interval: 5m0s
    query_ready_num_days: 0
    index_gateway_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: ""
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
          min_period: 100ms
          max_period: 10s
          max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
      log_gateway_requests: false
    use_boltdb_shipper_as_backup: false
    ingestername: 18b4ed97d8b9
    mode: RW
    ingesterdbretainperiod: 20m0s
    enable_postings_cache: false
chunk_store_config:
  chunk_cache_config:
    enable_fifocache: true
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 10
      writeback_buffer: 10000
      writeback_size_limit: 1GB
    memcached:
      expiration: 0s
      batch_size: 1024
      parallelism: 100
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      ttl: 1h0m0s
    fifocache:
      max_size_bytes: 1GB
      max_size_items: 0
      ttl: 1h0m0s
      validity: 0s
      size: 0
      purgeinterval: 0s
    prefix: store.chunks-cache.
    async_cache_write_back_concurrency: 16
    async_cache_write_back_buffer_size: 500
  chunk_cache_config_l2:
    enable_fifocache: false
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 10
      writeback_buffer: 10000
      writeback_size_limit: 1GB
    memcached:
      expiration: 0s
      batch_size: 1024
      parallelism: 100
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      ttl: 1h0m0s
    fifocache:
      max_size_bytes: 1GB
      max_size_items: 0
      ttl: 1h0m0s
      validity: 0s
      size: 0
      purgeinterval: 0s
    prefix: experimental.store.chunks-cache-l2.
    async_cache_write_back_concurrency: 16
    async_cache_write_back_buffer_size: 500
  write_dedupe_cache_config:
    enable_fifocache: false
    default_validity: 1h0m0s
    background:
      writeback_goroutines: 10
      writeback_buffer: 10000
      writeback_size_limit: 1GB
    memcached:
      expiration: 0s
      batch_size: 1024
      parallelism: 100
    memcached_client:
      host: ""
      service: memcached
      addresses: ""
      timeout: 100ms
      max_idle_conns: 16
      max_item_size: 0
      update_interval: 1m0s
      consistent_hash: true
      circuit_breaker_consecutive_failures: 10
      circuit_breaker_timeout: 10s
      circuit_breaker_interval: 10s
    redis:
      endpoint: ""
      master_name: ""
      timeout: 500ms
      expiration: 0s
      db: 0
      pool_size: 0
      username: ""
      password: ""
      tls_enabled: false
      tls_insecure_skip_verify: false
      idle_timeout: 0s
      max_connection_age: 0s
      route_randomly: false
    embedded_cache:
      max_size_mb: 100
      ttl: 1h0m0s
    fifocache:
      max_size_bytes: 1GB
      max_size_items: 0
      ttl: 1h0m0s
      validity: 0s
      size: 0
      purgeinterval: 0s
    prefix: store.index-cache-write.
    async_cache_write_back_concurrency: 16
    async_cache_write_back_buffer_size: 500
  l2_chunk_cache_handoff: 0s
  cache_lookups_older_than: 0s
  max_look_back_period: 0s
schema_config:
  configs:
  - from: "2020-05-15"
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: index_
      period: 1d
      tags: {}
    chunks:
      prefix: ""
      period: 0s
      tags: {}
    row_shards: 16
compactor:
  working_directory: /loki/compactor
  shared_store: s3
  shared_store_key_prefix: index/
  compaction_interval: 10m0s
  apply_retention_interval: 0s
  retention_enabled: false
  retention_delete_delay: 2h0m0s
  retention_delete_worker_count: 150
  retention_table_timeout: 0s
  delete_request_store: ""
  delete_batch_size: 70
  delete_request_cancel_period: 24h0m0s
  delete_max_interval: 0s
  max_compaction_parallelism: 1
  upload_parallelism: 10
  compactor_ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    tokens_file_path: ""
    zone_awareness_enabled: false
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_availability_zone: ""
    instance_enable_ipv6: false
  _: false
  tables_to_compact: 0
  skip_latest_n_tables: 0
  deletion_mode: ""
compactor_grpc_client:
  max_recv_msg_size: 104857600
  max_send_msg_size: 104857600
  grpc_compression: ""
  rate_limit: 0
  rate_limit_burst: 0
  backoff_on_ratelimits: false
  backoff_config:
    min_period: 100ms
    max_period: 10s
    max_retries: 10
  initial_stream_window_size: 63KiB1023B
  initial_connection_window_size: 63KiB1023B
  tls_enabled: false
  tls_cert_path: ""
  tls_key_path: ""
  tls_ca_path: ""
  tls_server_name: ""
  tls_insecure_skip_verify: false
  tls_cipher_suites: ""
  tls_min_version: ""
  connect_timeout: 5s
  connect_backoff_base_delay: 1s
  connect_backoff_max_delay: 5s
limits_config:
  ingestion_rate_strategy: global
  ingestion_rate_mb: 50
  ingestion_burst_size_mb: 100
  max_label_name_length: 10240
  max_label_value_length: 20480
  max_label_names_per_series: 300
  reject_old_samples: true
  reject_old_samples_max_age: 1w
  creation_grace_period: 10m
  enforce_metric_name: false
  max_line_size: 0B
  max_line_size_truncate: false
  increment_duplicate_timestamp: false
  max_streams_per_user: 0
  max_global_streams_per_user: 5000
  unordered_writes: true
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 15MB
  max_chunks_per_query: 2000000
  max_query_series: 500
  max_query_lookback: 0s
  max_query_length: 30d1h
  max_query_range: 0s
  max_query_parallelism: 32
  tsdb_max_query_parallelism: 512
  tsdb_max_bytes_per_shard: 600MB
  cardinality_limit: 100000
  max_streams_matchers_per_query: 1000
  max_concurrent_tail_requests: 10
  max_entries_limit_per_query: 1000
  max_cache_freshness_per_query: 10m
  max_stats_cache_freshness: 0s
  max_queriers_per_tenant: 0
  query_ready_index_num_days: 0
  query_timeout: 5m50s
  split_queries_by_interval: 1d
  min_sharding_lookback: 0s
  max_query_bytes_read: 0B
  max_querier_bytes_read: 0B
  volume_enabled: false
  volume_max_series: 1000
  ruler_evaluation_delay_duration: 0s
  ruler_max_rules_per_rule_group: 0
  ruler_max_rule_groups_per_tenant: 0
  ruler_alertmanager_config: null
  ruler_tenant_shard_size: 0
  ruler_remote_write_disabled: false
  ruler_remote_write_url: ""
  ruler_remote_write_timeout: 0s
  ruler_remote_write_headers: {}
  ruler_remote_write_queue_capacity: 0
  ruler_remote_write_queue_min_shards: 0
  ruler_remote_write_queue_max_shards: 0
  ruler_remote_write_queue_max_samples_per_send: 0
  ruler_remote_write_queue_batch_send_deadline: 0s
  ruler_remote_write_queue_min_backoff: 0s
  ruler_remote_write_queue_max_backoff: 0s
  ruler_remote_write_queue_retry_on_ratelimit: false
  ruler_remote_write_sigv4_config: null
  ruler_remote_evaluation_timeout: 0s
  ruler_remote_evaluation_max_response_size: 0
  deletion_mode: filter-and-delete
  retention_period: 90d
  per_tenant_override_config: ""
  per_tenant_override_period: 10s
  allow_deletes: false
  shard_streams:
    enabled: false
    logging_enabled: false
    desired_rate: 3MB
  index_gateway_shard_size: 1
frontend_worker:
  frontend_address: ""
  scheduler_address: ""
  dns_lookup_duration: 3s
  parallelism: 10
  match_max_concurrent: true
  id: ""
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
    grpc_compression: ""
    rate_limit: 0
    rate_limit_burst: 0
    backoff_on_ratelimits: false
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10
    initial_stream_window_size: 63KiB1023B
    initial_connection_window_size: 63KiB1023B
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    connect_timeout: 5s
    connect_backoff_base_delay: 1s
    connect_backoff_max_delay: 5s
table_manager:
  throughput_updates_disabled: false
  retention_deletes_enabled: false
  retention_period: 0s
  poll_interval: 2m0s
  creation_grace_period: 10m0s
  index_tables_provisioning:
    enable_ondemand_throughput_mode: false
    provisioned_write_throughput: 1000
    provisioned_read_throughput: 300
    write_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    read_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    enable_inactive_throughput_on_demand_mode: false
    inactive_write_throughput: 1
    inactive_read_throughput: 300
    inactive_write_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    inactive_read_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    inactive_write_scale_lastn: 4
    inactive_read_scale_lastn: 4
  chunk_tables_provisioning:
    enable_ondemand_throughput_mode: false
    provisioned_write_throughput: 1000
    provisioned_read_throughput: 300
    write_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    read_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    enable_inactive_throughput_on_demand_mode: false
    inactive_write_throughput: 1
    inactive_read_throughput: 300
    inactive_write_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    inactive_read_scale:
      enabled: false
      role_arn: ""
      min_capacity: 3000
      max_capacity: 6000
      out_cooldown: 1800
      in_cooldown: 1800
      target: 80
    inactive_write_scale_lastn: 4
    inactive_read_scale_lastn: 4
memberlist:
  node_name: ""
  randomize_node_name: true
  stream_timeout: 10s
  retransmit_factor: 4
  pull_push_interval: 30s
  gossip_interval: 200ms
  gossip_nodes: 3
  gossip_to_dead_nodes_time: 30s
  dead_node_reclaim_time: 0s
  compression_enabled: true
  advertise_addr: ""
  advertise_port: 7946
  cluster_label: ""
  cluster_label_verification_disabled: false
  join_members: []
  min_join_backoff: 1s
  max_join_backoff: 1m0s
  max_join_retries: 10
  abort_if_cluster_join_fails: false
  rejoin_interval: 0s
  left_ingesters_timeout: 5m0s
  leave_timeout: 20s
  message_history_buffer_bytes: 0
  bind_addr: []
  bind_port: 7946
  packet_dial_timeout: 2s
  packet_write_timeout: 5s
  tls_enabled: false
  tls_cert_path: ""
  tls_key_path: ""
  tls_ca_path: ""
  tls_server_name: ""
  tls_insecure_skip_verify: false
  tls_cipher_suites: ""
  tls_min_version: ""
runtime_config:
  period: 10s
  file: ""
tracing:
  enabled: true
analytics:
  reporting_enabled: false
  usage_stats_url: https://stats.grafana.org/loki-usage-report
legacy_read_target: true
common:
  path_prefix: /loki
  storage:
    s3:
      s3: ""
      s3forcepathstyle: false
      bucketnames: ""
      endpoint: ""
      region: ""
      access_key_id: ""
      secret_access_key: ""
      session_token: ""
      insecure: false
      sse_encryption: false
      http_config:
        timeout: 0s
        idle_conn_timeout: 1m30s
        response_header_timeout: 0s
        insecure_skip_verify: false
        ca_file: ""
      signature_version: v4
      storage_class: STANDARD
      sse:
        type: ""
        kms_key_id: ""
        kms_encryption_context: ""
      backoff_config:
        min_period: 100ms
        max_period: 3s
        max_retries: 5
    gcs:
      bucket_name: ""
      service_account: ""
      chunk_buffer_size: 0
      request_timeout: 0s
      enable_opencensus: true
      enable_http2: true
    azure:
      environment: AzureGlobal
      account_name: ""
      account_key: ""
      container_name: loki
      endpoint_suffix: ""
      use_managed_identity: false
      use_federated_token: false
      user_assigned_id: ""
      use_service_principal: false
      client_id: ""
      client_secret: ""
      tenant_id: ""
      chunk_delimiter: '-'
      download_buffer_size: 512000
      upload_buffer_size: 256000
      upload_buffer_count: 1
      request_timeout: 30s
      max_retries: 5
      min_retry_delay: 10ms
      max_retry_delay: 500ms
    alibabacloud:
      bucket: ""
      endpoint: ""
      access_key_id: ""
      secret_access_key: ""
    bos:
      bucket_name: ""
      endpoint: bj.bcebos.com
      access_key_id: ""
      secret_access_key: ""
    swift:
      auth_version: 0
      auth_url: ""
      internal: false
      username: ""
      user_domain_name: ""
      user_domain_id: ""
      user_id: ""
      password: ""
      domain_id: ""
      domain_name: ""
      project_id: ""
      project_name: ""
      project_domain_id: ""
      project_domain_name: ""
      region_name: ""
      container_name: ""
      max_retries: 3
      connect_timeout: 10s
      request_timeout: 5s
    filesystem:
      chunks_directory: ""
      rules_directory: ""
    hedging:
      at: 0s
      up_to: 2
      max_per_second: 5
    cos:
      forcepathstyle: false
      bucketnames: ""
      endpoint: ""
      region: ""
      access_key_id: ""
      secret_access_key: ""
      http_config:
        idle_conn_timeout: 1m30s
        response_header_timeout: 0s
      backoff_config:
        min_period: 100ms
        max_period: 3s
        max_retries: 5
      api_key: ""
      service_instance_id: ""
      auth_endpoint: https://iam.cloud.ibm.com/identity/token
      cr_token_file_path: ""
      trusted_profile_name: ""
      trusted_profile_id: ""
  persist_tokens: false
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
      prefix: collectors/
      consul:
        host: localhost:8500
        acl_token: ""
        http_client_timeout: 20s
        consistent_reads: false
        watch_rate_limit: 1
        watch_burst_size: 1
        cas_retry_delay: 1s
      etcd:
        endpoints: []
        dial_timeout: 10s
        max_retries: 10
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        username: ""
        password: ""
      multi:
        primary: ""
        secondary: ""
        mirror_enabled: false
        mirror_timeout: 2s
    heartbeat_period: 15s
    heartbeat_timeout: 1m0s
    tokens_file_path: ""
    zone_awareness_enabled: false
    instance_id: 18b4ed97d8b9
    instance_interface_names:
    - eth0
    - lo
    instance_port: 0
    instance_addr: 127.0.0.1
    instance_availability_zone: ""
    instance_enable_ipv6: false
  instance_interface_names:
  - eth0
  instance_addr: ""
  compactor_address: ""
  compactor_grpc_address: ""
shutdown_delay: 0s

Hope it helps.

9

Also, if I check the rules at /loki/api/v1/rules there shows

no rule groups found

and rules are in Grafana:

image
image2490×664 38.5 KB

But still, this doesn’t answer why Grafana requires Loki to have shared storage for the ruler to enable alert history, when the rules are managed by Grafana itself :woman_shrugging:

10

  1. I still don’t see retention enabled in your configuration. At this point I am not sure why compactor is trying to apply retention when it’s not enabled. Are you using simple scalable mode?

  2. This could be why Grafana is trying to hit your ruler:

Screenshot 2024-05-29 at 5.34.48 PM
Screenshot 2024-05-29 at 5.34.48 PM1082×252 14.7 KB

This is part of the data source configuration. If you enable this whenever you open up Grafana’s alert view it’ll try to connect to Loki ruler. You can disable it then Grafana should not try to connect to your ruler anymore.

11

  1. Me neither, i’m quite lost. We enabled retention once to test and disabled it afterwards. There are no files inside the /loki/compactor/retention folder nor inside /loki/compactor/deletion folder.

  2. That section is enabled as well:

12

  1. If you are not using a production cluster, I’d say shut down all Loki containers, make sure configuration everywhere is refreshed, then try again. If you are on a production cluster you can copy the configuration to a dev/test cluster (change S3 bucket of course) and see if you can replicate the issue.

  2. What I meant is you want to disable that option. If you disable it Grafana won’t interact with Loki ruler. If you enable it then Grafana will try to list rules from Loki ruler in UI.