Help a beginner with grafana,mimir,loki,prome in k8

Hello Community! :slight_smile:

I’m starting to learn the grafana-stack bit by bit and i’m currently trying to get all the peaces to fit.

But i have some things that i don’t manage to understand. Sorry for the stupid questions…
What i’m trying to accomplish is to install prometheus,loki,mimir,grafana in a kubernetes kluster.

Everything almost work as i think it would but have a few obstacles. There is a lot info to take in so i hope someone could point me in the correct direction of thinking.

The thing for now is that i don’t understand some stuff about Mimir and a question for Loki.

  1. Where does the metrics go? Is it locally for this “test-mimir” in a minIO container? I guess you can send the metrics to a object-storage. If i want the send the metrics “on-prem” does that work, or does it only work that you have to have a service like s3/minio that cost money? Is MinIO just a forwarder to S3?

  2. Is the grafana agent only for cloud?

  3. How do i include local logs for loki. I can only see {filename=/var/log/pods…} I tried to add the following in promtail, values.yaml

# local machine logs
- job_name: system
  - targets:
      - localhost
      job: varlogs
      __path__: /var/log/*log
  1. This current Mimir-setup is not HA i guess?

  2. I can login to grafana and i can search logs in loki and i can also receive metrics. But i have problem to get Mimir to work. My plan is that i want multiple k8 clusters that sends metrics and logs to a central mimir/loki, but this is just for me to understand for now. I have added the mimir dashboard ID: 16026. But i can’t see any incoming data. Maybe someone could help me. I want the metrics from my prometheus to come in to Mimir. In config i added as you can see bellow: remote_write. I wonder if i’m using the wrong endpoint. I also tried to add the prometheus endpoint: http://mimir-nginx.observability.svc:80/prometheus. But still no data in mimir/writes - Dashboard.

When i’m starting up the mimir-test with docker-compose. i can see that the datasource that was added points to: http://load-balancer:9009/prometheus and remote_write: http://load-balancer:9009/api/v1/push. Here i see data comes in.

And i found this when i searched for remote_rewrite:
Use the cluster.local service url:
And force Mimir to listen on 9009 port with http_listen_port: 9009

Very confused what it should be -_-

simple map.

My current setup:

running locally with minikube:

kubectl create ns observability
helm install loki . --namespace observability
helm install kube-prometheus-stack . --namespace observability
helm -n observability install mimir grafana/mimir-distributed
kubectl port-forward service/kube-prometheus-stack-grafana -n observability 3000:80

added in config: values.yaml → kube-prometheus-stack

      - url: http://mimir-nginx.observability.svc:80/api/v1/push

      - job_name: prometheus
        honor_labels: true
          - targets: ["localhost:9090"]

    - name: Loki
      type: loki
      url: http://loki.observability.svc.cluster.local:3100
      access: proxy  
            - datasourceUid: tempo
              matcherRegex: ((\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\
              name: TraceID
              url: $${__value.raw}
          maxLines: 1000
      version: 1

Question regarding minIO loki.

If you want to run scalable version of loki then it requires a compatible s3 object store such as s3 or a self-hosted store as minIO. Monolithic is only local filesystem.

But i thought that loki only use push, does loki save log files aswell?

I guess this is the same for Mimir and metrics. But here you pull metrics.

if i enable remoteWriteDashboards: false to true i can see writes from mimir with that dashboard instead. But not with the official mimir write dashboard. - prometheus remotewrite dashboard - mimir dashboard