Inquiry on How to Query Data from AWS S3 Remotely in Loki

Grafana Loki
1

I am currently using the loki-stack helm-charts and querying data from it as a data source in Grafana. However, in Grafana, I can only query data for a specific period, and I’m unable to retrieve historical data.

  • I have configured share_object to S3, and the index information is being stored there. I’ve confirmed the S3 object.

What I want is to store Loki index information in S3 and query this data in Grafana to view historical data. I understand that Loki supports this.

Below is the code for the Loki-stack helm-chart values. Is there something I’m doing incorrectly?

loki:
enabled: true
isDefault: true
url: http://{{(include “loki.serviceName” .)}}:{{ .Values.loki.service.port }}
readinessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
livenessProbe:
httpGet:
path: /ready
port: http-metrics
initialDelaySeconds: 45
datasource:
jsonData: “{}”
uid: “”
serviceAccountName: loki-sa
serviceAccount:
create: true
name: loki-sa
annotations:
eks.amazonaws.com/role-arn: “arn:aws:iam::xxxxxx:role/lgtm-mimir”
auth_enabled: false
commonConfig:
path_prefix: /var/loki
replication_factor: 1
compactor:
apply_retention_interval: 1h
compaction_interval: 720m
retention_delete_worker_count: 500
retention_enabled: true
shared_store: s3
working_directory: /data/compactor
config:
schema_config:
configs:
- from: 2020-05-15
store: boltdb-shipper
object_store: s3
schema: v11
index:
period: 24h
prefix: loki_index_
storage_config:
aws:
region: ap-northeast-2
bucketnames: lgtm-loki-storage-xxxxxx-ap-northeast-2
s3forcepathstyle: false
# s3forcepathstyle: true ← This is the main culprit; comment it out ? -? Loki with S3 not able to push chunks on S3 bucket with compactor error "NoSuchKeys" · Issue #7024 · grafana/loki · GitHub
boltdb_shipper:
shared_store: s3

promtail:
enabled: true
config:
logLevel: info
serverPort: 3101
clients:
- url: http://{{ .Release.Name }}:3100/loki/api/v1/push

grafana:
enabled: true
sidecar:
datasources:
label: “”
labelValue: “”
enabled: true
maxLines: 1000
image:
tag: 8.3.5
users:
default_theme: dark

prometheus:
enabled: false
isDefault: false
url: http://{{ include “prometheus.fullname” .}}:{{ .Values.prometheus.server.service.servicePort }}{{ .Values.prometheus.server.prefixURL }}
datasource:
jsonData: “{}”

2

Your configuration is not formatted, so it’s a bit difficult to read. But I am pretty sure the aws under storage_config should be changed to s3. Or you can use a common block to define S3 storage for all components.

Here is what our configuration looks like specific to S3:

common:
  storage:
    s3:
      bucketnames: <aws_bucketname>
      region: <aws_region>
      endpoint: "<_aws_s3_endpoint>"
      sse_encryption: true
      s3forcepathstyle: true