Grafana and Jaeger - 502 bad gateway - connection refused

  • What Grafana version and what operating system are you using?

  • v8.2.0 (d7f71e9eae) running in docker on Windows 10.

  • What are you trying to achieve?

Connect Jaeger as a data source after the initial install of Grafana.

  • How are you trying to achieve it?

I am filling out the Jaeger data source dialog. I enter http://localhost:16686 and click on Save and Test.

  • What happened?

The following is displayed in the data source screen. Jaeger: Bad Gateway. 502. Bad Gateway shown in browser

In the docker image for grafana I can see a connection refused message

t=2021-10-11T01:34:53+0000 lvl=eror msg=“Data proxy error” logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/services remote_addr=172.17.0.1 referer=http://localhost:3000/datasources/edit/xCVWnQv7k error=“http: proxy error: dial tcp 127.0.0.1:16686: connect: connection refused”

Both Jaeger and Grafana are running in docker on the same machine as I am on. All localhost. I can connect to Jaegar in the browser and I can connect to Jaegar via powershell without error on the same computer hosting the docker images.

Invoke-WebRequest -uri http://127.0.0.1:16686

I can also connect using c#

        WebClient wc = new WebClient();
        string data = wc.DownloadString("http://localhost:16686");

But Grafana does not seem to be able to connect to the Jaegar instance. Grafana is running on localhost on port 3000

  • What did you expect to happen?

It should connect without error.

  • Can you copy/paste the configuration(s) that you are having problems with?

No configuration changes made except trying to add the data source.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

t=2021-10-11T01:34:53+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/services remote_addr=172.17.0.1 referer=http://localhost:3000/datasources/edit/xCVWnQv7k error="http: proxy error: dial tcp 127.0.0.1:16686: connect: connection refused"

t=2021-10-11T01:34:53+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/services status=502 remote_addr=172.17.0.1 time_ms=1 size=0 referer=http://localhost:3000/datasources/edit/xCVWnQv7k

t=2021-10-11T01:41:58+0000 lvl=eror msg="Data proxy error" logger=data-proxy-log userId=1 orgId=1 uname=admin path=/api/datasources/proxy/3/api/services remote_addr=172.17.0.1 referer=http://localhost:3000/datasources/edit/xCVWnQv7k error="http: proxy error: dial tcp 127.0.0.1:16686: connect: connection refused"

t=2021-10-11T01:41:58+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/api/datasources/proxy/3/api/services status=502 remote_addr=172.17.0.1 time_ms=1 size=0 referer=http://localhost:3000/datasources/edit/xCVWnQv7k

  • Did you follow any online instructions? If so, what is the URL?

2 days and no replies - any idea how I debug this issue?

localhost in the container is not localhost on the host. Each container has own network namespace (so also localhost) + it probably more complicated from the network perspective, because you are running Docker on windows - and that Docker is running on Linux VM usually. Generally, don’t use localhost, but hostname/IP (not 127.X.X.X - that’s localhost IP).