I have installed loki-stack using attached values.yaml loki:
I am able to view logs from pods, containers. I have installed tempo using helm install tempo grafana/tempo
Able to view traces as well but not able to corelate the traces to logs. Trying to add derived fields but when clicked on Test in loki getting → “Unable to connect with Loki. Please check the server logs for more details”.
[enter image description here](
![loki-datasource-conf|518x500](upload://iYqgA6GFOJ9OZbZkTDOj8G7Mcxy.png)
```loki:
auth_enabled: false
commonConfig:
path_prefix: /var/loki
replication_factor: 1
compactor:
apply_retention_interval: 1h
compaction_interval: 5m
retention_delete_worker_count: 500
retention_enabled: true
shared_store: s3
working_directory: /data/compactor
config:
schema_config:
configs:
- from: 2020-05-15
store: boltdb-shipper
object_store: s3
schema: v11
index:
period: 24h
prefix: loki_index_
storage_config:
aws:
region: us-east-1
bucketnames: grafana-loki-bucket
s3forcepathstyle: false
shared_store: s3
cache_ttl: 24h
serviceAccount:
create: true
name: loki-sa
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::xxxxxx:role/grafana-loki-role"
write:
replicas: 2
read:
replicas: 1
promtail:
enabled: true
config:
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /promtail/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
pipeline_stages:
- docker: {}
- cri: {}
- json:
expressions:
severity: level
timestamp: ts
traceId: trace_id
- replace:
source: traceId
target_label: traceID
regex: "(.*)"
- drop:
source: traceId
relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
action: replace
target_label: container
- source_labels: [__meta_kubernetes_pod_uid]
action: replace
target_label: pod_uid
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
extraVolumes:
- name: promtail-positions
emptyDir: {}
extraVolumeMounts:
- name: promtail-positions
mountPath: /promtail
grafana:
enabled: true
)
grafana svc has been exposed as a Load balancer and the loki datasource was added by default and there isn't an option change the loki URL.
checking the loki svc, grafana and loki pods but didn't find any issues