Cannot get Grafana to recognize Mimir monolitic instance, bare metal

Hello all,

I’m getting started with Mimir and I’m trying to connect the following running on the same machine:

And yet when I try to setup a new datasource pointing to my new Mimir instance I get an error saying:

401 Unauthorized - There was an error returned querying the Prometheus API.

My settings (only showing overrides):

Prometheus server URL: http://localhost:9009/prometheus
Prometheus type: Mimir
Mimir version: > 2.9x
Cache level: None
Method: POST

I went back and forth on the documentation and I don’t think I missed anything. Gladly I can paste my Mimir configuration here for review.


Can you please post your configs for prom and mimir

Prom 9090
Mimir 9009

Sure thing, here are my settings (some values changed to protect the innocent):


multitenancy_enabled: true

  enabled: false

  http_listen_port: 9009
  log_level: error

  # Configure the server to allow messages up to 100MB.
  grpc_server_max_recv_msg_size: 104857600
  grpc_server_max_send_msg_size: 104857600
  grpc_server_max_concurrent_streams: 1000

  compactor_blocks_retention_period: 3m
  out_of_order_time_window: 10m

  enable_otel_metadata_translation: true

      store: memberlist
    health_check_ingesters: true

      - vmx1.ZZZ
    instance_addr: 172.26.yy.xx

    min_ready_duration: 0s
    final_sleep: 0s
    num_tokens: 512

      store: inmemory
    replication_factor: 1

    dir: /scratch/grafana/tmp/mimir/tsdb

    sync_dir: /scratch/grafana/tmp/mimir/tsdb-sync

    dir: /scratch/grafana/mimir/data/tsdb

  data_dir: /scratch/grafana/tmp/mimir/compactor

  backend: local
    directory: /scratch/grafana/tmp/mimir/rules


app_mode = production
instance_name = ${HOSTNAME}
data = /scratch/grafana

protocol = http

domain =

reporting_enabled = false
check_for_updates = true

admin_user = admin
admin_password = xxxx


versions_to_keep = 10
min_refresh_interval = 1m




name = GitHub
icon = github
enabled = false
allow_sign_up = true
auto_login = false
client_id = XXX
client_secret = YYYY
scopes = user:email,read:org
auth_url =
token_url =
api_url =

enabled = true
config_file = /etc/grafana/ldap.toml
allow_sign_up = true

enabled = true
host =
from_address =



log_rotate = true

max_days = 5

enabled = false

:grin: how about prometheus config and what are you scraping?

OK, that’s the part where I think I’m lost. I will have components pushing/ remote writing to Mimir. So that is not there yet.

I’m confused because I tough that I could have Mimir as the central Prometheus data source.

1 Like

you are not alone. I also assumed you assumed that. It is a bit confusing to almost everyone that starts out with this stack.

how will you push stuff to mimir ?

prometheus = monitoring and can be setup to push monitored data to mimir
mimir = prometheus long term storage tsdb etc

how will you push stuff to mimir ?
prometheus = monitoring and can be setup to push monitored data to mimir
mimir = prometheus long term storage tsdb etc

I want to have applications pushing data to Mimir (for long term storage, aggregation). The applications will not use Prometheus but OpenTelemetry (but they have a way to push compatible metrics).
Then Grafana will be able to display time series, etc. from Mimir stored data.

I was also checking Grafana agent flow, but do you need to have now 3 components to forward the data? And that still doesn’t solve the issue of Grafana not being able to talk to Mimir.

1 Like

I added the HTTP Header as I have multi-tenant enabled but the real issue what the URL. So I changed the port to use the default, on Mimir, and also the end URL on the Grafana datasource:


And it works now!

1 Like