-
What Grafana version and what operating system are you using?
11.6.3 via Amazon Linux 2 in AWS ECS -
What are you trying to achieve?
Query Cloudwatch logs and metrics from various AWS accounts within my organization from a single Grafana instance running in an ECS cluster in its own AWS account.
For the first step, I just try to get the Cloudwatch logs and metrics from the same AWS account where Grafana is running.
- How are you trying to achieve it?
Assuming a role within the Cloudwatch data source.
- I created a role “GrafanaCloudwatchAccessRole” with a) permissions to access Cloudwatch and b) a trust policy that allows the ecs task to assume the role.
- The ECS task role has permissions to assume the “GrafanaCloudwatchAccessRole”
- Via the Grafana UI of the Cloudwatch data source I add the arn to the “GrafanaCloudwatchAccessRole” role.
-
What happened?
UI showing error: 504 Gateway Time-out -
What did you expect to happen?
That the connection to Cloudwatch can established by assuming the role. -
Can you copy/paste the configuration(s) that you are having problems with?
data “aws_iam_policy_document” “grafana_cloudwatch_access_assume” {
statement {
actions = [“sts:AssumeRole”]
effect = “Allow”
principals {
type = “AWS”
identifiers = [“arn:aws:iam::<account_id_a>:role/GATSHA-shared-EcsTask-grafana”]
}
}
}
resource “aws_iam_role” “grafana_cloudwatch_access” {
name = “GrafanaCloudwatchAccessRole”
path = “/${var.coordinates.scope}/monitoring/”
assume_role_policy = data.aws_iam_policy_document.grafana_cloudwatch_access_assume.json
permissions_boundary = “arn:aws:iam::${var.coordinates.account_id}:policy/ScopePermissionBoundary”
}
resource “aws_iam_policy” “grafana_cloudwatch_access” {
name = “${var.coordinates.scope}-grafana-cloudwatch-access-policy”
description = “Policy to allow Grafana in the shared monitoring account to access CloudWatch.”
policy = jsonencode({
Version = “2012-10-17”,
Statement = [
{
Sid = “AllowReadingMetricsFromCloudWatch”,
Effect = “Allow”,
Action = [
“cloudwatch:DescribeAlarmsForMetric”,
“cloudwatch:DescribeAlarmHistory”,
“cloudwatch:DescribeAlarms”,
“cloudwatch:ListMetrics”,
“cloudwatch:GetMetricStatistics”,
“cloudwatch:GetMetricData”,
“cloudwatch:GetInsightRuleReport”
],
Resource = “"
},
{
Sid = “AllowReadingLogsFromCloudWatch”,
Effect = “Allow”,
Action = [
“logs:DescribeLogGroups”,
“logs:GetLogGroupFields”,
“logs:StartQuery”,
“logs:StopQuery”,
“logs:GetQueryResults”,
“logs:GetLogEvents”
],
Resource = "”
},
{
Sid = “AllowReadingResourceMetricsFromPerformanceInsights”,
Effect = “Allow”,
Action = “pi:GetResourceMetrics”,
Resource = “"
},
{
Sid = “AllowReadingTagsInstancesRegionsFromEC2”,
Effect = “Allow”,
Action = [
“ec2:DescribeTags”,
“ec2:DescribeInstances”,
“ec2:DescribeRegions”
],
Resource = "”
},
{
Sid = “AllowReadingResourcesForTags”,
Effect = “Allow”,
Action = “tag:GetResources”,
Resource = “"
},
{
Sid = “AllowReadingOAMResources”,
Effect = “Allow”,
Action = [
“oam:ListSinks”,
“oam:ListAttachedLinks”
],
Resource = "”
}
]
})
}
resource “aws_iam_role_policy_attachment” “grafana_cloudwatch_access” {
role = aws_iam_role.grafana_cloudwatch_access.name
policy_arn = aws_iam_policy.grafana_cloudwatch_access.arn
}
and here the policy for the ECS task role to allow assuming the above role:
resource “aws_iam_policy” “assume_cross_account_cloudwatch” {
name = “${var.coordinates.scope}-grafana-cloudwatch-assume-policy”
description = “Policy to allow Grafana to assume a cross-account role for CloudWatch access.”
policy = jsonencode({
Version = “2012-10-17”,
Statement = [
{
Sid = “AllowAssumingCrossAccountCloudwatchRole”,
Effect = “Allow”,
Action = [
“sts:AssumeRole”
],
Resource = “arn:aws:iam::<account_id_a>:role/GATSHA/monitoring/GrafanaCloudwatchAccessRole”,
}
]
})
}
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
In the UI after adding the role arn: 504 Gateway Timeout
grafana logs:
logger=tsdb.cloudwatch endpoint=callResource pluginId=cloudwatch dsName=cloudwatch dsUID=ber70rddn9y4ga uname=admin t=2025-07-07T14:19:15.612800886Z level=error msg=“Error handling resource request” error=“error getting accounts for current user or role: ListSinks error: RequestError: send request failed\ncaused by: Post "https ://sts.amazonaws.com/": dial tcp 52.94.139.12:443: i/o timeout”
logger=tsdb.cloudwatch endpoint=callResource pluginId=cloudwatch dsName=cloudwatch dsUID=ber70rddn9y4ga uname=admin t=2025-07-07T14:19:15.613006569Z level=error msg="Failed to get regions: " error=“RequestError: send request failed\ncaused by: Post "https ://sts.amazonaws.com/": dial tcp 52.94.139.12:443: i/o timeout”
- Did you follow any online instructions? If so, what is the URL?
Amazon CloudWatch data source | Grafana documentation
What am I missing to connect to Cloudwatch? FYI, when adding the policy directly to the task role, I can access Cloudwatch. But this is not a solution to query Cloudwatch in multiple accounts. Also, within the Grafana container I can use the aws cli to assume the role and query Cloudwatch.
