Old loki logs not showing in Grafana after Loki Helm upgrade

Grafana Loki
1

hello community,

the last few days I have been working on upgrading our loki to a newer version in our EKS. We keep the the logs in a s3 bucket and I can see the new logs being created there. The logs are still there. Nevertheless when using grafana to see the logs I can not see the logs older that the upgrade time. Any pointers?
This is my values.yaml config:

loki:
  auth_enabled: false
  structuredConfig:
    chunk_store_config:
      max_look_back_period: 4320h
    table_manager:
      retention_deletes_enabled: true
      retention_period: 4320h
  storage:
    type: s3
    s3:
      region: REGION
      s3: s3://REGION/BUCKET
      s3ForcePathStyle: true
      insecure: true
    bucketNames:
      chunks: BUCKET
      rules: BUCKET
      admin: BUCKET
    boltdb_shipper:
      active_index_directory: /data/loki/index
      cache_ttl: 24h
      shared_store: s3
      cache_location: /data/loki/index_cache
  containerSecurityContext:
    readOnlyRootFilesystem: false
  ingester:
    chunk_block_size: 262144
    chunk_idle_period: 3m
    chunk_retain_period: 1m
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
    max_transfer_retries: 0
  schemaConfig:
    configs:
      - from: 2021-10-31
        store: boltdb-shipper
        object_store: s3
        schema: v11
        index:
          prefix: index_
          period: 24h
  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 4320h

serviceAccount:
  annotations:
    "eks.amazonaws.com/role-arn": "arn:aws:iam::XXXXXXXX:role/loki-role"

singleBinary:
  replicas: 1
  persistence:
#    enabled: false
    size: 5Gi


provisioner:
  enabled: false

enterprise:
  enabled: false

monitoring:
  selfMonitoring:
    enabled: false
    grafanaAgent:
      installOperator: false
  lokiCanary:
    enabled: false
  read:
    enabled: false
  write:
    enabled: false
  tableManager:
    enabled: false

gateway:
  enabled: false

test:
  enabled: false
2

Don’t see anything obvious. Couple of questions:

  1. What version did you upgrade from and to?
  2. What changed in configuration?
  3. Error messages in reader or writer?
3

Thank you Tony,

  1. The version upgrade is from version 2.7.0 (helm chart 2.3.0) to version 2.8.3 (helm chart 5.9.2)
  2. I am trying to match the old version the vest I can the old config will look like this:
auth_enabled: false
chunk_store_config:
  max_look_back_period: 4320h
compactor:
  shared_store: filesystem
  working_directory: /data/loki/boltdb-shipper-compactor
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 4320h
schema_config:
  configs:
  - from: "2021-10-31"
    index:
      period: 24h
      prefix: index_
    object_store: s3
    schema: v11
    store: boltdb-shipper
server:
  http_listen_port: 3100
storage_config:
  aws:
    s3: s3://REGION/BUCKET
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /data/loki/index
    cache_location: /data/loki/index_cache
    cache_ttl: 24h
    shared_store: s3
  filesystem:
    directory: /data/loki/chunks
table_manager:
  retention_deletes_enabled: true
  retention_period: 4320h

And the new one:

auth_enabled: false
chunk_store_config:
  max_look_back_period: 4320h
common:
  compactor_address: 'loki'
  path_prefix: /var/loki
  replication_factor: 3
  storage:
    s3:
      bucketnames: BUCKET
      insecure: true
      region: REGION
      s3: s3://REGION/BUCKET
      s3forcepathstyle: true
frontend:
  scheduler_address: ""
frontend_worker:
  scheduler_address: ""
index_gateway:
  mode: ring
ingester:
  chunk_block_size: 262144
  chunk_idle_period: 3m
  chunk_retain_period: 1m
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  max_transfer_retries: 0
limits_config:
  enforce_metric_name: false
  max_cache_freshness_per_query: 10m
  reject_old_samples: true
  reject_old_samples_max_age: 4320h
  split_queries_by_interval: 15m
memberlist:
  join_members:
  - loki-memberlist
query_range:
  align_queries_with_step: true
ruler:
  storage:
    s3:
      bucketnames: ruler
      insecure: true
      region: REGION
      s3: s3://REGION/BUCKET
      s3forcepathstyle: true
    type: s3
runtime_config:
  file: /etc/loki/runtime-config/runtime-config.yaml
schema_config:
  configs:
  - from: "2021-10-31"
    index:
      period: 24h
      prefix: index_
    object_store: s3
    schema: v11
    store: boltdb-shipper
server:
  grpc_listen_port: 9095
  http_listen_port: 3100
storage_config:
  hedging:
    at: 250ms
    max_per_second: 20
    up_to: 3
table_manager:
  retention_deletes_enabled: true
  retention_period: 4320h
  1. I get this error:
    level=error ts=2023-08-01T10:00:20.896343627Z caller=index_set.go:285 table-name=index_19526 msg=“sync failed, retrying it” err=“WebIdentityErr: failed to retrieve credentials\ncaused by: RequestError: send request failed\ncaused by: Post "http://sts.amazonaws.com/\”: dial tcp 209.54.180.124:80: connect: connection refused"

It seems like access issue, but Loki pushes new logs in the same s3.

4

The error could be because you set insecure to true. I’d try changing your storage configuration a little:

common:
  storage:
    s3:
      bucketnames: BUCKET
      region: REGION
      s3forcepathstyle: true
5

Thank you Tony.
It is working with insecure: true.
But for some reason it stated working when I removed the service account annotation for access to the s3 bucket. Loki is using ec2 access to s3 instead.

6

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.