Getting started: prometheus and grafana inside same docker instance

Grafana Configuration
1

Ultimately I’ll be standing up a Prometheus / Grafana inside a single AWS Fargate instance. I’m working on it inside a Docker instance on my desktop. I found something weird I’m trying to understand.

I configured my datasource like this:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: direct
    orgId: 1
    url: http://localhost:9090
    isDefault: true

And I start the instance with:

docker run -d \
	--name prometheus \
	--hostname prometheus \
	-p 8110:22 \
#		-p 9090:9090 \
	-p 9091:9091 \
	prometheus:latest

I have prometheus on 9090 and grafana on 9091. I wasn’t originally doing the -p 9090:9090, and Grafana was reporting No metrics. I decided to break the problem into two and tried to attach to Prometheus’s built in graphing, which is when I added the port map for Prometheus, and suddenly Grafana reports data.

So I’m confused. Does the URL used for the datasource need to be externally resolvable? Is it actually being hit by my own browser or something along those lines? When I configure this for my AWS instance, do I need to make Prometheus accessible and use a datasource URL that will resolve?

Can someone explain why port-mapping the Prometheus port suddenly exposed my data?

(I suspect if I knew how Grafana worked under the hood, the answer would be obvious. I think I’ll go digging for that.)

-Joe

2

Not sure I understand fully understand the problem, but let me know what I got wrong and I’ll update my answer. When setting up your Prometheus data source, you can choose if you want to make the request from your browser or from the server (access mode). If both Grafana and Prometheus are running in separate containers, without any network linking them, you need to set the access mode to browser.

If you just want to try it out, you can check out the environment for our tutorials, which uses Docker Compose to link the containers together.

3

They’re in the same container.

So, this is the important part?

datasources:
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy

My datasource provisioning file looks like this:

datasources:
  - name: Prometheus
    type: prometheus
    access: direct
    orgId: 1
    url: http://localhost:9090
    isDefault: true

I should change access to proxy and my problem goes away?

4

If both are in the same container, and both are exposed on the host, it shouldn’t matter which one you pick.

5

Under Fargate, AWS’s container system, “exposed to the host” doesn’t mean anything.

I switched it to proxy and removed the port mapping for Prometheus (no longer exposing it), and I still get data on my desktop. I’m now in the process of pushing to AWS and testing.

6

Okay, that was what was required. I now have graphs. I switched access from direct to proxy in my datasource, and Grafana is happy referencing Prometheus via localhost, Prometheus NOT exposed to the outside world.

1 Like
7

I’m glad it worked out for you!