# IMDSv2 is not working from docker

**URL:** <https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944>\
**Category:** Configuration\
**Tags:** aws, auth, cloudwatch\
**Created:** [May 19, 2022, 12:47pm UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944 "2022-05-19T12:47:37Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![yaron](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/yaron/32/23326_2.png) [@yaron](https://community.grafana.com/u/yaron)\
**Post date:** [May 19, 2022, 12:47pm UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944/1 "2022-05-19T12:47:37Z")

</div>

Hi, I’m using Grafana docker hosted on an EC2 instance with IMDSv2 enabled.

While testing with the following `curl` commands everything looks fine (the token is fake):

```bash
$ TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 56 100 56 0 0 65268 0 --:--:-- --:--:-- --:--:-- 56000
$ curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/
* Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/meta-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> X-aws-ec2-metadata-token: AQAEAOpL35wR9KpFB7rRxVy7GhnB2mIBGPXNR5tBP4c-f7YWiL1yXA==
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21593
< Content-Type: text/plain
< Accept-Ranges: none
< Last-Modified: Thu, 19 May 2022 11:39:54 GMT
< Content-Length: 337
< Date: Thu, 19 May 2022 12:33:04 GMT
< Server: EC2ws
< Connection: close
< 
ami-id
ami-launch-index
…

```

Now when I’m trying to add CloudWatch data source to Grafana (inside the docker container) the following message appears (Text is attached as alternative caption for the image):

 ![image](https://us1.discourse-cdn.com/grafana/original/3X/c/0/c01fe8ca53afb38327bf32e363275f3303945c85.png "1. CloudWatch metrics query failed: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: EC2MetadataError: failed to make EC2Metadata request status code: 401, request id: 2. CloudWatch logs query failed: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: EC2MetadataError: failed to make EC2Metadata request status code: 401, request id:")

What am I doing wrong?

Grafana version: 8.5.2  
Relevant Attached Policy (although it doesn’t matter): CloudWatchReadOnlyAccess

UPDATE: I’ve discovered that’s it’s not working due to network restrictions.

---

<div class="post-metadata">

**Author:** ![anon70503803](https://avatars.discourse-cdn.com/v4/letter/a/8797f3/32.png) [@anon70503803](https://community.grafana.com/u/anon70503803)\
**Post date:** [May 26, 2022, 6:57pm UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944/2 "2022-05-26T18:57:40Z")

</div>

can you elaborate in case this thread help someone else in the future? thx 👍

---

<div class="post-metadata">

**Author:** ![yaron](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/yaron/32/23326_2.png) [@yaron](https://community.grafana.com/u/yaron)\
**Post date:** [May 27, 2022, 6:13am UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944/3 "2022-05-27T06:13:10Z")

</div>

Sure, well, docker allows all sorts of networking modes, internal, bridge and host.

When using bridge or host, the container is exposed to everything the machine is exposed to including the internal AWS API (IMDS/v2) via 169.254.169.254.

So, when selecting an internal network for Grafana (although there’s an exposed port) the container cannot reach IMDS, one possible solution is create another network and attach the Grafana container to it or switching to bridge or host mode entirely, in my case it’s not recommended as it exposes promtail and loki without any moderation.

Hoping my explanation makes sense.

---

<div class="post-metadata">

**Author:** ![connorsmith](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/connorsmith/32/37173_2.png) [@connorsmith](https://community.grafana.com/u/connorsmith)\
**Post date:** [August 21, 2023, 6:17am UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944/4 "2023-08-21T06:17:52Z")

</div>

This Stack Overflow question goes into more detail about this and, in my opinion, a better solution.

IMDSv2 by default has a hop limit of 1. Running in a container makes the hop count 2.  
Instead of changing your Docker configuration, you should increase the EC2 instances hop limit.

[amazon web services - Using IMDS (v2) with token inside docker on EC2 or ECS - Stack Overflow](https://stackoverflow.com/questions/71884350/using-imds-v2-with-token-inside-docker-on-ec2-or-ecs)

---

<div class="post-metadata">

**Author:** ![yaron](https://sea2.discourse-cdn.com/grafana/user_avatar/community.grafana.com/yaron/32/23326_2.png) [@yaron](https://community.grafana.com/u/yaron)\
**Post date:** [August 24, 2023, 11:39am UTC](https://community.grafana.com/t/imdsv2-is-not-working-from-docker/65944/5 "2023-08-24T11:39:46Z")

</div>

BTW there’s no Web UI option, the only way we managed to change this setting was via CLI (CloudShell or whatever you prefer).