Hello,
we have deployed the loki on our microk8s clusters with the below give s3 storage configuration and facing the below error. Could someone help me. what was the issue and how to proceed further.
Values.file
storage:
bucketNames:
chunks: loki-con/loki-chunks
ruler: loki-con/loki-ruler
type: s3
s3:
s3: s3:////loki-con
endpoint: https://XXXXXXXXXXXXXXXXXXXX
region:
secretAccessKey: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
accessKeyId: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
s3ForcePathStyle: false
insecure: false
http_config:
insecure_skip_verify: true
schemaConfig:
configs:
- from: 2024-04-01
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
Error which im getting:
level=info ts=2024-08-03T11:56:18.846348957Z caller=flush.go:196 component=ingester msg=“flushing stream” user=fake fp=51a0e12ef2f93627 immediate=false num_chunks=4 labels=“{app="csi-cinder-nodeplugin", container="cinder-csi-plugin", filename="/var/log/pods/kube-system_csi-cinder-nodeplugin-7zbxs_5ad821e1-a109-4ecb-9239-c136ab6178d0/cinder-csi-plugin/0.log", job="kube-system/csi-cinder-nodeplugin", namespace="kube-system", node_name="k8s-master-0", pod="csi-cinder-nodeplugin-7zbxs", service_name="csi-cinder-nodeplugin", stream="stderr"}”
level=error ts=2024-08-03T11:56:18.848717121Z caller=flush.go:178 component=ingester loop=7 org_id=fake msg=“failed to flush” retries=9 err=“failed to flush chunks: store put chunk: SerializationError: failed to unmarshal error message\n\tstatus code: 404, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000 3c 21 44 4f 43 54 59 50 45 20 48 54 4d 4c 20 50 |..|\n00000040 3c 74 69 74 6c 65 3e 34 30 34 20 4e 6f 74 20 46 |404 Not F|\n00000050 6f 75 6e 64 3c 2f 74 69 74 6c 65 3e 0a 3c 2f 68 |ound.</h|\n00000060 65 61 64 3e 3c 62 6f 64 79 3e 0a 3c 68 31 3e 4e |ead>.
N|\n00000070 6f 74 20 46 6f 75 6e 64 3c 2f 68 31 3e 0a 3c 70 |ot Found
.<p|\n00000080 3e 54 68 65 20 72 65 71 75 65 73 74 65 64 20 55 |>The requested U|\n00000090 52 4c 20 77 61 73 20 6e 6f 74 20 66 6f 75 6e 64 |RL was not found|\n000000a0 20 6f 6e 20 74 68 69 73 20 73 65 72 76 65 72 2e | on this server.|\n000000b0 3c 2f 70 3e 0a 3c 68 72 3e 0a 3c 61 64 64 72 65 |..<addre|\n000000c0 73 73 3e 41 70 61 63 68 65 2f 32 2e 34 2e 34 31 |ss>Apache/2.4.41|\n000000d0 20 28 55 62 75 6e 74 75 29 20 53 65 72 76 65 72 | (Ubuntu) Server|\n000000e0 20 61 74 20 72 61 64 6f 73 67 77 2e 61 70 69 2e | at radosgw.api.|\n000000f0 64 6b 2d 62 6c 70 30 31 2e 68 63 70 2e 64 73 76 |dk-blp01.hcp.dsv|\n00000100 2e 63 6f 6d 20 50 6f 72 74 20 34 34 33 3c 2f 61 |.com Port 443</a|\n00000110 64 64 72 65 73 73 3e 0a 3c 2f 62 6f 64 79 3e 3c |ddress>.<|\n00000120 2f 68 74 6d 6c 3e 0a |/html>.|\n\ncaused by: expected element type but have , num_chunks: 4, labels: {app="csi-cinder-nodeplugin", container="cinder-csi-plugin", filename="/var/log/pods/kube-system_csi-cinder-nodeplugin-7zbxs_5ad821e1-a109-4ecb-9239-c136ab6178d0/cinder-csi-plugin/0.log", job="kube-system/csi-cinder-nodeplugin", namespace="kube-system", node_name="k8s-master-0", pod="csi-cinder-nodeplugin-7zbxs", service_name="csi-cinder-nodeplugin", stream="stderr"}”
level=error ts=2024-08-03T11:56:18.848779126Z caller=flush.go:156 component=ingester loop=7 org_id=fake msg=“failed to flush” err=“terminated after 10 retries”
level=info ts=2024-08-03T11:56:18.848845152Z caller=flush.go:196 component=ingester msg=“flushing stream” user=fake fp=51a0e12ef2f93627 immediate=false num_chunks=4 labels=“{app="csi-cinder-nodeplugin", container="cinder-csi-plugin", filename="/var/log/pods/kube-system_csi-cinder-nodeplugin-7zbxs_5ad821e1-a109-4ecb-9239-c136ab6178d0/cinder-csi-plugin/0.log", job="kube-system/csi-cinder-nodeplugin", namespace="kube-system", node_name="k8s-master-0", pod="csi-cinder-nodeplugin-7zbxs", service_name="csi-cinder-nodeplugin", stream="stderr"}”
level=error ts=2024-08-03T11:56:18.850907261Z caller=flush.go:178 component=ingester loop=7 org_id=fake msg=“failed to flush” retries=0 err=“failed to flush chunks: store put chunk: SerializationError: failed to unmarshal error message\n\tstatus code: 404, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000 3c 21 44 4f 43 54 59 50 45 20 48 54 4d 4c 20 50 |..|\n00000040 3c 74 69 74 6c 65 3e 34 30 34 20 4e 6f 74 20 46 |404 Not F|\n00000050 6f 75 6e 64 3c 2f 74 69 74 6c 65 3e 0a 3c 2f 68 |ound.</h|\n00000060 65 61 64 3e 3c 62 6f 64 79 3e 0a 3c 68 31 3e 4e |ead>.
N|\n00000070 6f 74 20 46 6f 75 6e 64 3c 2f 68 31 3e 0a 3c 70 |ot Found
.<p|\n00000080 3e 54 68 65 20 72 65 71 75 65 73 74 65 64 20 55 |>The requested U|\n00000090 52 4c 20 77 61 73 20 6e 6f 74 20 66 6f 75 6e 64 |RL was not found|\n000000a0 20 6f 6e 20 74 68 69 73 20 73 65 72 76 65 72 2e | on this server.|\n000000b0 3c 2f 70 3e 0a 3c 68 72 3e 0a 3c 61 64 64 72 65 |..<addre|\n000000c0 73 73 3e 41 70 61 63 68 65 2f 32 2e 34 2e 34 31 |ss>Apache/2.4.41|\n000000d0 20 28 55 62 75 6e 74 75 29 20 53 65 72 76 65 72 | (Ubuntu) Server|\n000000e0 20 61 74 20 72 61 64 6f 73 67 77 2e 61 70 69 2e | at radosgw.api.|\n000000f0 64 6b 2d 62 6c 70 30 31 2e 68 63 70 2e 64 73 76 |dk-blp01.hcp.dsv|\n00000100 2e 63 6f 6d 20 50 6f 72 74 20 34 34 33 3c 2f 61 |.com Port 443</a|\n00000110 64 64 72 65 73 73 3e 0a 3c 2f 62 6f 64 79 3e 3c |ddress>.<|\n00000120 2f 68 74 6d 6c 3e 0a |/html>.|\n\ncaused by: expected element type but have , num_chunks: 4, labels: {app="csi-cinder-nodeplugin", container="cinder-csi-plugin", filename="/var/log/pods/kube-system_csi-cinder-nodeplugin-7zbxs_5ad821e1-a109-4ecb-9239-c136ab6178d0/cinder-csi-plugin/0.log", job="kube-system/csi-cinder-nodeplugin", namespace="kube-system", node_name="k8s-master-0", pod="csi-cinder-nodeplugin-7zbxs", service_name="csi-cinder-nodeplugin", stream="stderr"}”