Random HTTP 405 - Method not Allowed

  • What Grafana version and what operating system are you using?
    Grafana 7.5.7 + Debian 10 Buster inside a Docker container

  • What are you trying to achieve?
    Containerized Grafana instance, able to spin up at a moment’s notice

  • How are you trying to achieve it?
    Running a grafana instance inside a docker container with file-provisioned dashboards and a Prometheus data source - Available over network on the Docker host server.

  • What happened?
    Accessing the Grafana instance, every provisioned item gets imported correctly, however, dashboards only show interrupted data segments -

When I take the widget’s PromQL query and plug it straight into the prometheus data source, it returns a continual data graph, so the data is physically alright inside the prometheus instance.

Investigating network requests my browser makes as I load the dashboard shows a few odd POST requests ending with HTTP 405 - Method not Allowed. the URL at fault here is /api/datasources/proxy/1/api/v1/series, but when I go over to edit a widget, I also get the error from /api/datasources/proxy/1/api/v1/metadata

Grafana’s logs also show no apparent source of the issue, only thing I get is a log like this:

t=2021-06-25T13:35:17+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/1/api/v1/metadata status=405 remote_addr=my.ip time_ms=1 size=19 referer="https://domain.tld/d/000000033/apache?orgId=1&refresh=1m&editPanel=5"

I am accessing the Grafana instance through the master admin account, and proxy the requests through an Nginx instance that acts as a SSL terminator and HTTP authentication provider.

To proxy the requests, I am using the built-in proxy_pass functionality to a localhost-bound port opened by the docker container.

I tried having the prometheus datasource instance bound on the docker network range (172.17.0.0/16), then rebound it on the server’s public WAN… Neither fixed the issue.

  • What did you expect to happen?
    Graphs to render correctly and fully

  • Can you copy/paste the configuration(s) that you are having problems with?
    Prometheus provisioning configuration:

apiVersion: 2

datasources:
    - version: 4
      name: Prometheus
      type: prometheus
      access: proxy
      url: http://ip-address:9091
      jsonData:
          httpMethod: 'POST' #Note - I added this as a debug meassure, to find out if a POST would stop the HTTP 405 errors from happening, initially I had the default GET. Neither helps with my issue 

Grafana configuration (Stripped of comments and unused configuration parameters / sections):

app_mode = production
[paths]
data = /var/lib/grafana/
logs = /var/log/grafana/
plugins = /var/lib/grafana/plugins
[server]
protocol = http
http_port = 3001
domain = *domain*
root_url = https://*url*
[security]
admin_user = *username*
admin_password = *password*
disable_brute_force_login_protection = true
[smtp]
enabled = true
  • Did you follow any online instructions? If so, what is the URL?
    Nope, set up a grafana instance in the same manner as other instances we run and work correctly, only difference is it being inside docker container.
1 Like

I have the same behaviour here, too. I’m running a simple docker-compose (grafana/grafana-enterprise) version 8.2.2 setup with nginx reverse proxy in front. Everything seems to work from the browsers view but I still get some errors in the logs:

a.b.c.d - - [22/Oct/2021:13:45:06 +0200] "POST /api/datasources/proxy/1/api/v1/query_exemplars HTTP/2.0" 405 19 "https://grafana.kretschmann.software/d/w80xjLd7z/bintra?orgId=1&refresh=10s" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.71 Safari/537.36"

which gives in the docker.compose log something like:

grafana_1 | t=2021-10-22T11:44:46+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/1/api/v1/query_exemplars status=405 remote_addr=a.b.c.d time_ms=39 size=19 referer="https://grafana.example.com/d/w80xjLd7z/bintra?orgId=1&refresh=10s"

So, how to get rid of that stuff?

1 Like

I have the same behavior:

method=POST path=/api/datasources/proxy/96/api/v1/query_exemplars status=405

5 log lines at once each 5 seconds.

docker swarm / traefik / prometeus