Getting 422 Unprocessable Entity when adding prometheus datasource in Docker

Grafana Configuration
, ,
1
  • What Grafana version and what operating system are you using?

Grafana v11.3.0 (d9455ff7db) docker oss image on an Almalinux 9.4 system

  • What are you trying to achieve?

I’m trying to create a prometheus datasource. I have the prometheus instance in the same docker environment, and i am able to “ping” the prometheus host by name from the grafana instance. But when i try to create and save my datasource for prometheus i get the following error:

422 Unprocessable Entity - There was an error returned querying the Prometheus API.

  • How are you trying to achieve it?

I’m clicking on the create data source and giving it a name “prometheus” and server_url of “http://prometheus:9090” and click save.

  • What happened?

It creates the datasource, but it’s not valid and i get the following error:

422 Unprocessable Entity - There was an error returned querying the Prometheus API.

  • What did you expect to happen?

I expected to get the same response as the tutorial docker container:

Successfully queried the Prometheus API.

  • Can you copy/paste the configuration(s) that you are having problems with?
    my compose file is:
networks:
  grafana-net:

volumes:
  prometheus-data: {}
  grafana-data: {}


services:
  prometheus:
    image: prom/prometheus:v2.49.0
    container_name: prometheus
    ports:
      - 9090:9090
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.agent.path=/prometheus'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
      - '--enable-feature=agent'
    volumes:
      - ./config:/etc/prometheus
      - prometheus-data:/prometheus
    restart: unless-stopped
    networks:
      - grafana-net

  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - '3000:3000'
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped
    #password: root123
    networks:
      - grafana-net

My prometheus config is:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"

    static_configs:
      - targets: ["localhost:9090"]
  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

I only have the 422 error in the Grafana UI as described above.

  • Did you follow any online instructions? If so, what is the URL?

I’ve followed the docs on installing Grafana via Docker: Run Grafana Docker image | Grafana documentation

i also got the tutorial up and running to compare

2

OMG! could this be that the prometheus that’s getting loaded is the Agent vs the server??? The Tutorial endpoint is VERY different from mine. The instance in my docker environment is the Prometheus Agent. Currently, pursuing that…

3

That was it. I remvoed the following and now get successfully connected when i save.