Unable to use AWS CloudWatch as a Datasource with AWS IAM role

Grafana Cloudwatch
1
  1. CloudWatch metrics query failed: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get “http://169.254.169.254/latest/meta-data/iam/security-credentials/”: dial tcp 169.254.169.254:80: connect: connection refused

Getting the above error when trying to create AWS cloudwatch as a Datasource?

EC2 IAM Role:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “AllowReadingMetricsFromCloudWatch”,
“Effect”: “Allow”,
“Action”: [
“cloudwatch:DescribeAlarmsForMetric”,
“cloudwatch:DescribeAlarmHistory”,
“cloudwatch:DescribeAlarms”,
“cloudwatch:ListMetrics”,
“cloudwatch:GetMetricData”,
“cloudwatch:GetInsightRuleReport”
],
“Resource”: “"
},
{
“Sid”: “AllowReadingLogsFromCloudWatch”,
“Effect”: “Allow”,
“Action”: [
“logs:DescribeLogGroups”,
“logs:GetLogGroupFields”,
“logs:StartQuery”,
“logs:StopQuery”,
“logs:GetQueryResults”,
“logs:GetLogEvents”
],
“Resource”: "”
},
{
“Sid”: “AllowReadingTagsInstancesRegionsFromEC2”,
“Effect”: “Allow”,
“Action”: [
“ec2:DescribeTags”,
“ec2:DescribeInstances”,
“ec2:DescribeRegions”
],
“Resource”: “"
},
{
“Sid”: “AllowReadingResourcesForTags3”,
“Effect”: “Allow”,
“Action”: “tag:GetResources”,
“Resource”: "”
},
{
“Effect”: “Allow”,
“Action”: [
“xray:BatchGetTraces”,
“xray:GetTraceSummaries”,
“xray:GetTraceGraph”,
“xray:GetGroups”,
“xray:GetTimeSeriesServiceStatistics”,
“xray:GetInsightSummaries”,
“xray:GetInsight”,
“xray:GetServiceGraph”,
“ec2:DescribeRegions”
],
“Resource”: “*”
}
]
}

Followed article: Amazon CloudWatch data source | Grafana documentation

I am successfully able to execute:
[root@ip-172-31-42-199 ~]# curl -vv 169.254.169.254/latest/meta-data/iam/security-credentials/

  • Trying 169.254.169.254:80…
  • Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)

GET /latest/meta-data/iam/security-credentials/ HTTP/1.1
Host: 169.254.169.254
User-Agent: curl/7.87.0
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 200 OK
    < Content-Type: text/plain
    < Accept-Ranges: none
    < Last-Modified: Wed, 08 Feb 2023 04:50:07 GMT
    < Content-Length: 11
    < Date: Wed, 08 Feb 2023 04:52:21 GMT
    < Server: EC2ws
    < Connection: close
    <
  • Closing connection 0

Still not sure why it is not working

2

That look like you didn’t attach an IAM role to an EC2 instance correctly. Check with your AWS support/doc, e. g. IAM roles for Amazon EC2 - Amazon Elastic Compute Cloud