Logs no longer returned once in s3

Grafana Loki
1

My loki deployment just stopped returning logs from s3 today.

Here’s the config we’ve been running for month:

affinity: {}
alerting_groups: []
annotations: {}
client: {}
config:
  auth_enabled: false
  chunk_store_config:
    max_look_back_period: 5040h
  compactor:
    shared_store: filesystem
    working_directory: /data/loki/boltdb-shipper-compactor
  ingester:
    chunk_block_size: 262144
    chunk_idle_period: 3m
    chunk_retain_period: 1m
    lifecycler:
      ring:
        kvstore:
          store: inmemory
        replication_factor: 1
    max_transfer_retries: 0
    wal:
      dir: /data/loki/wal
  limits_config:
    enforce_metric_name: false
    reject_old_samples: true
    reject_old_samples_max_age: 168h
  schema_config:
    configs:
    - from: "2021-12-30"
      index:
        period: 24h
        prefix: index_
      object_store: s3
      schema: v11
      store: boltdb-shipper
  server:
    http_listen_port: 2355
  storage_config:
    aws:
      s3: s3://XXXXX+XXXXXX@my-region/such-a-cool-s3-bucket
    boltdb_shipper:
      active_index_directory: /data/loki/boltdb-shipper-active
      cache_location: /data/loki/boltdb-shipper-cache
      cache_ttl: 24h
      shared_store: filesystem
    filesystem:
      directory: /data/loki/chunks
  table_manager:
    retention_deletes_enabled: true
    retention_period: 5040h
env: []
extraArgs: {}
extraContainers: []
extraPorts: []
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: loki-prod-k8s
image:
  pullPolicy: IfNotPresent
  repository: grafana/loki
  tag: 2.4.1
ingress:
  annotations: {}
  enabled: false
  hosts:
  - host: chart-example.local
    paths: []
  tls: []
initContainers: []
livenessProbe:
  httpGet:
    path: /ready
    port: http-metrics
  initialDelaySeconds: 45
networkPolicy:
  enabled: false
nodeSelector: {}
persistence:
  accessModes:
  - ReadWriteOnce
  annotations: {}
  enabled: false
  size: 100Gi
podAnnotations:
  prometheus.io/port: http-metrics
  prometheus.io/scrape: "true"
podDisruptionBudget: {}
podLabels: {}
podManagementPolicy: OrderedReady
rbac:
  create: true
  pspEnabled: true
readinessProbe:
  httpGet:
    path: /ready
    port: http-metrics
  initialDelaySeconds: 45
replicas: 1
resources: {}
securityContext:
  fsGroup: 10001
  runAsGroup: 10001
  runAsNonRoot: true
  runAsUser: 10001
service:
  annotations: {}
  labels: {}
  name: loki-service
  nodePort: null
  port: 2355
  targetPort: http-metrics
  type: ClusterIP
serviceAccount:
  annotations: {}
  automountServiceAccountToken: true
  create: true
  name: null
serviceMonitor:
  additionalLabels: {}
  annotations: {}
  enabled: false
  interval: ""
terminationGracePeriodSeconds: 4800
tolerations: []
tracing:
  jaegerAgentHost: null
updateStrategy:
  type: RollingUpdate

When I’m running a query, I don’t get any logs past 24hrs, but I can see the logs are being written to s3 and there are no errors in the logs.

Our loki version is: 2.4.1 - but we also tried 2.8.4
This is on k8s and deployed with helm

2

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.