Cannot get Grafana to recognize Mimir monolitic instance, bare metal

Grafana Data Links
,
1

Hello all,

I’m getting started with Mimir and I’m trying to connect the following running on the same machine:

And yet when I try to setup a new datasource pointing to my new Mimir instance I get an error saying:

401 Unauthorized - There was an error returned querying the Prometheus API.

My settings (only showing overrides):

Prometheus server URL: http://localhost:9009/prometheus
Prometheus type: Mimir
Mimir version: > 2.9x
Cache level: None
Method: POST

I went back and forth on the documentation and I don’t think I missed anything. Gladly I can paste my Mimir configuration here for review.

Thanks.

2

Can you please post your configs for prom and mimir

Prom 9090
Mimir 9009

3

Sure thing, here are my settings (some values changed to protect the innocent):

/etc/mimir/config.yml

multitenancy_enabled: true

usage_stats:
  enabled: false

server:
  http_listen_port: 9009
  log_level: error

  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

limits:
  compactor_blocks_retention_period: 3m
  out_of_order_time_window: 10m

api:
  enable_otel_metadata_translation: true

distributor:
  ring:
    kvstore:
      store: memberlist
  pool:
    health_check_ingesters: true

ingester:
  ring:
    instance_interface_names:
      - vmx1.ZZZ
    instance_addr: 172.26.yy.xx

    min_ready_duration: 0s
    final_sleep: 0s
    num_tokens: 512

    kvstore:
      store: inmemory
    replication_factor: 1

blocks_storage:
  tsdb:
    dir: /scratch/grafana/tmp/mimir/tsdb

  bucket_store:
    sync_dir: /scratch/grafana/tmp/mimir/tsdb-sync

  filesystem:
    dir: /scratch/grafana/mimir/data/tsdb

compactor:
  data_dir: /scratch/grafana/tmp/mimir/compactor

ruler_storage:
  backend: local
  local:
    directory: /scratch/grafana/tmp/mimir/rules

/etc/grafana/grafana.ini

app_mode = production
instance_name = ${HOSTNAME}
[paths]
data = /scratch/grafana

[server]
protocol = http

domain = grafana.example.com

[analytics]
reporting_enabled = false
check_for_updates = true

[security]
admin_user = admin
admin_password = xxxx

[snapshots]

[dashboards]
versions_to_keep = 10
min_refresh_interval = 1m

[users]

[auth]

[auth.github]

name = GitHub
icon = github
enabled = false
allow_sign_up = true
auto_login = false
client_id = XXX
client_secret = YYYY
scopes = user:email,read:org
auth_url = https://ghe-myorg.example.com/login/oauth/authorize
token_url = https://ghe-myorg.example.com/login/oauth/access_token
api_url = https://ghe-myorg.example.com/api/v3/user

[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

[smtp]
enabled = true
host = mail.example.com:25
from_address = myorgtechnologyus@example.com

[log]
[log.console]


[log.file]


log_rotate = true


max_days = 5

[alerting]
enabled = false
4

:grin: how about prometheus config and what are you scraping?

5

OK, that’s the part where I think I’m lost. I will have components pushing/ remote writing to Mimir. So that is not there yet.

I’m confused because I tough that I could have Mimir as the central Prometheus data source.

1 Like
6

you are not alone. I also assumed you assumed that. It is a bit confusing to almost everyone that starts out with this stack.

how will you push stuff to mimir ?

prometheus = monitoring and can be setup to push monitored data to mimir
mimir = prometheus long term storage tsdb etc

7

how will you push stuff to mimir ?
prometheus = monitoring and can be setup to push monitored data to mimir
mimir = prometheus long term storage tsdb etc

I want to have applications pushing data to Mimir (for long term storage, aggregation). The applications will not use Prometheus but OpenTelemetry (but they have a way to push compatible metrics).
Then Grafana will be able to display time series, etc. from Mimir stored data.

I was also checking Grafana agent flow, but do you need to have now 3 components to forward the data? And that still doesn’t solve the issue of Grafana not being able to talk to Mimir.

1 Like
8

I added the HTTP Header as I have multi-tenant enabled but the real issue what the URL. So I changed the port to use the default, on Mimir, and also the end URL on the Grafana datasource:

http://myserver.example:8080/prometheus

And it works now!

1 Like