Hello Community! 
I’m starting to learn the grafana-stack bit by bit and i’m currently trying to get all the peaces to fit.
But i have some things that i don’t manage to understand. Sorry for the stupid questions…
What i’m trying to accomplish is to install prometheus,loki,mimir,grafana in a kubernetes kluster.
Everything almost work as i think it would but have a few obstacles. There is a lot info to take in so i hope someone could point me in the correct direction of thinking.
The thing for now is that i don’t understand some stuff about Mimir and a question for Loki.
-
Where does the metrics go? Is it locally for this “test-mimir” in a minIO container? I guess you can send the metrics to a object-storage. If i want the send the metrics “on-prem” does that work, or does it only work that you have to have a service like s3/minio that cost money? Is MinIO just a forwarder to S3?
-
Is the grafana agent only for cloud?
-
How do i include local logs for loki. I can only see {filename=/var/log/pods…} I tried to add the following in promtail, values.yaml
# local machine logs
scrape_configs:
- job_name: system
static_configs:
- targets:
- localhost
labels:
job: varlogs
__path__: /var/log/*log
-
This current Mimir-setup is not HA i guess?
-
I can login to grafana and i can search logs in loki and i can also receive metrics. But i have problem to get Mimir to work. My plan is that i want multiple k8 clusters that sends metrics and logs to a central mimir/loki, but this is just for me to understand for now. I have added the mimir dashboard ID: 16026. But i can’t see any incoming data. Maybe someone could help me. I want the metrics from my prometheus to come in to Mimir. In config i added as you can see bellow: remote_write. I wonder if i’m using the wrong endpoint. I also tried to add the prometheus endpoint: http://mimir-nginx.observability.svc:80/prometheus. But still no data in mimir/writes - Dashboard.
When i’m starting up the mimir-test with docker-compose. i can see that the datasource that was added points to: http://load-balancer:9009/prometheus and remote_write: http://load-balancer:9009/api/v1/push. Here i see data comes in.
And i found this when i searched for remote_rewrite:
Use the cluster.local service url:
http://mimir.namespace.svc.cluster.local:9009/api/v1/push
And force Mimir to listen on 9009 port with http_listen_port: 9009
Very confused what it should be -_-
simple map.
My current setup:
running locally with minikube:
kubectl create ns observability
helm install loki . --namespace observability
helm install kube-prometheus-stack . --namespace observability
helm -n observability install mimir grafana/mimir-distributed
kubectl port-forward service/kube-prometheus-stack-grafana -n observability 3000:80
added in config: values.yaml → kube-prometheus-stack
remote_write:
- url: http://mimir-nginx.observability.svc:80/api/v1/push
scrape_configs:
- job_name: prometheus
honor_labels: true
static_configs:
- targets: ["localhost:9090"]
additionalDataSources:
- name: Loki
type: loki
url: http://loki.observability.svc.cluster.local:3100
access: proxy
jsonData:
derivedFields:
- datasourceUid: tempo
matcherRegex: ((\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\
d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+)(\d+|[a-z]+))
name: TraceID
url: $${__value.raw}
maxLines: 1000
version: 1
