Loki SSD in AKS Storage Errors (the hostname became incomplete after some time)

Grafana Installation
1

  • What Grafana version and what operating system are you using?
    Latest HELM SSD

  • What are you trying to achieve?
    AKS SSD Helm Deployment

  • How are you trying to achieve it?
    HELM SSD

  • What happened?
    Loki containers are not respecting the blob account host name.

  • What did you expect to happen?
    calls to be made to the correct storage account

  • Can you copy/paste the configuration(s) that you are having problems with?

VALUES.YAML for the HELM
loki:
schemaConfig:
configs:
- from: “2020-12-11”
index:
period: 24h
prefix: index_
object_store: azure
schema: v11
store: tsdb
storage:
azure:
account_name: MYACCOUNTNAME
account_key: ‘XXXXX’
container_name: loki
request_timeout: 0
tsdb_shipper:
active_index_directory: /var/tsdb-index
cache_location: /var/tsdb-cache
cache_ttl: 24h
shared_store: azure

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

dial tcp: lookup .blob.core.windows.net: no such host

here you can see that the prefix (where the azure storage account name should be) is missing

  • Did you follow any online instructions? If so, what is the URL?

No, I have been unable to identify how this even happened in the first place.

2

some additional information here, I believe this state was achieved by the storage becoming maxxed out on the pvc, when the storage limit was reached that’s when this began.

I was able to get this working again by simply doing a helm upgrade with my previous values.yaml file - but I’m still unsure as to why/how this could have happened.