Issue immidately after running helm install with the loki stack repo in kubernetes

Grafana Loki
1

Hello all,

I’ve used the following command:
helm upgrade --install loki grafana/loki-stack --values ./loki-values.yaml -n logging

with the following values files

loki:
  enabled: true
  persistence:
    enabled: true
    storageClassName: longhorn
    size: 5Gi

promtail:
  enabled: true

grafana:
  enabled: true
  sidecar:
    datasources:
      enabled: true
  image:
    tag: 8.1.6

prometheus:
  enabled: false

When i launch grafana i get a 404 error, first initial login always fails, no dashboards exist at all, i can’t adjust user settings or change my password or create dashboards. Its a fresh install of this stack on a kuberetes k3s multinode cluster version v1.21.7.

Really wanted to try this out in my home lab. Any assistance on what I may have done wrong would be super helpful. I expected the out of the box install to at least function so I’m assuming I did something wrong.

I do have a previous installation of rancher monitoring setup just incase this is conflicting with anything but they are in different namespaces.

2

Hi @funkel1989 ,

are you using the “local” Grafana admin account to log in? I would assume the helm run would show the credentials. I have not used this helm chart… My first suspicion would be that the user you use does not have admin access. I could be wrong.

I would also look at the Grafana pods logs. iirc Grafana logs are pretty good at telling you what is not working as expected.

3

Hello @b0b . To confirm I am using the local grafana admin account that was created by the helm install using the password received from the secret created by the helm install. Neither the grafana container or the grafana datasources container are producing any logs. I find this odd but would be interested if you have any suggestions.

4

Hi @funkel1989 ,

I’m fairly new to Kubernetes and especially running Grafana on Kubernetes. I have no experience with k3s. So anything I write is mostly speculation…

In my testing I have done the following

Deployed Grafana with

My grafana/values.yaml

$ cat grafana/values.yaml
persistence:
  type: pvc
  enabled: true
  # storageClassName: default
  accessModes:
    - ReadWriteOnce
  size: 10Gi
  # annotations: {}
  finalizers:
    - kubernetes.io/pvc-protection
grafana.ini:
  paths:
    data: /var/lib/grafana/
    logs: /var/log/grafana
    plugins: /var/lib/grafana/plugins
    provisioning: /etc/grafana/provisioning
  analytics:
    check_for_updates: true
  users:
    viewers_can_edit: true
  log:
    mode: console
  grafana_net:
    url: https://grafana.net
  server:
    root_url: https://grafana.my.domain.com

Loki test deployment

My loki/values.yaml

$ cat loki/values.yaml
image:
  repository: grafana/loki
  tag: 2.3.0
  pullPolicy: IfNotPresent

Instead of Promtail I have used Grafana Agent. I have just applied plain text Kubernetes resource files for that.

I have worked on Kubernetes clusters with strict namespaces and network policies. I have left all that out here. This is just to give an idea of what has worked for me.

Hope that helps or at least gives some ideas of what to try.

I checked my Grafana pod and it is logging as expected.

For comparison, here is the full description of my Grafana pod

$ k describe pods  grafana-79c9f9c9b8-7gqgt -n grafana
Name:         grafana-79c9f9c9b8-7gqgt
Namespace:    grafana
Priority:     0
Node:        my-node
Start Time:   Sun, 21 Nov 2021 03:10:44 +0100
Labels:       app.kubernetes.io/instance=grafana
              app.kubernetes.io/name=grafana
              pod-template-hash=79c9f9c9b8
Annotations:  checksum/config: 15515d821eb302de3708075b9e0dfd28ffb40729c43b3c383d1e80d004d611a7
              checksum/dashboards-json-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
              checksum/sc-dashboard-provider-config: 01ba4719c80b6fe911b091a7c05124b64eeece964e09c058ef8f9805daca546b
              checksum/secret: 40b4f702267aeca530e46415edd07bfe51fe19453567c963f5fdc2e4a843bfd4
              kubernetes.io/psp: eks.privileged
Status:       Running
IP:           10.0.0.1
IPs:
  IP:           10.0.0.1
Controlled By:  ReplicaSet/grafana-79c9f9c9b8
Init Containers:
  init-chown-data:
    Container ID:  docker://8f4245e96c05043d80f3d63836709338a0218f22098604e302b921f5b8e7e648
    Image:         busybox:1.31.1
    Image ID:      docker-pullable://busybox@sha256:95cf004f559831017cdf4628aaf1bb30133677be8702a8c5f2994629f637a209
    Port:          <none>
    Host Port:     <none>
    Command:
      chown
      -R
      472:472
      /var/lib/grafana
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sun, 21 Nov 2021 03:29:15 +0100
      Finished:     Sun, 21 Nov 2021 03:29:15 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/lib/grafana from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from grafana-token-dq4cp (ro)
Containers:
  grafana:
    Container ID:   docker://ad93ee998edd08f073016a50ee4688a8c6ed06564182c3215a4c65b11289c945
    Image:          grafana/grafana:8.1.5
    Image ID:       docker-pullable://grafana/grafana@sha256:b7dd9cd0e59d107bbf8b62ded1860b888955d902a700e1997b1c9f211bcf8708
    Ports:          80/TCP, 3000/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 21 Nov 2021 03:29:25 +0100
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:3000/api/health delay=60s timeout=30s period=10s #success=1 #failure=10
    Readiness:      http-get http://:3000/api/health delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      GF_SECURITY_ADMIN_USER:      <set to the key 'admin-user' in secret 'grafana'>      Optional: false
      GF_SECURITY_ADMIN_PASSWORD:  <set to the key 'admin-password' in secret 'grafana'>  Optional: false
      GF_PATHS_DATA:               /var/lib/grafana/
      GF_PATHS_LOGS:               /var/log/grafana
      GF_PATHS_PLUGINS:            /var/lib/grafana/plugins
      GF_PATHS_PROVISIONING:       /etc/grafana/provisioning
    Mounts:
      /etc/grafana/grafana.ini from config (rw,path="grafana.ini")
      /var/lib/grafana from storage (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from grafana-token-dq4cp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      grafana
    Optional:  false
  storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  grafana
    ReadOnly:   false
  grafana-token-dq4cp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  grafana-token-dq4cp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.