I run Grafana v. 9.4.3 in an ECS cluster in AWS (Fargate).
I am trying to configure Cloudwatch access to a different AWS account using Roles and External IDs.
For that purpose I have created a role in the target AWS account, which has all required permissions Grafana requires to read Cloudwatch metrices and logs. That role can be assumed from the AWS account running Grafana using a specific External ID.
In The Grafana-AWS-account I have created a role allowing sts:AssumeRole of the relevant role in the target account and have attached that role as a TaskRole to the ECS task running Grafana. In Grafana itself I have configured the datasource to use ‘AWS SDK Default’ as the Authentication Provider, configured the relevant role for ‘Assume Role ARN’ and the relevant ‘External ID’.
When saving and testing that configuration, Grafan reports error:
'1. CloudWatch metrics query failed: NoCredentialProviders: no valid providers in chain caused by: EnvAccessKeyNotFound: failed to find credentials in the environment. SharedCredsLoad: failed to load profile, . EC2RoleRequestError: no EC2 instance role found caused by: RequestError: send request failed caused by: Get “http://169.254.169.254/latest/meta-data/iam/security-credentials/”: dial tcp 169.254.169.254:80: connect: invalid argument
I was expecting Grafana to be able to assume the correct role and read Cloudwatch data from the target AWS account.
Has anybody achieved to use only roles (no individual programmatic user account) to read Cloudwatch data in AWS account B from a Grafana running in a Fargate ECS task in AWS account A?
Thank you for your reply. Unfortunately, it does not help at all. Obviously, to me it does not really matter whether it is a ‘Grafana problem’ or an ‘AWS issue’ - the fact is that it have not yet managed to make it work.
To me Grafana’s error message seems to indicate it does not see/use any credentials it may receive from the Task role in AWS. But what do you mean by ‘your ECS task /execution role must be allowed to assume another role’? There are actually two roles attached to my task. I created (in AWS account A where Grafana runs) a Task role containing Action sts:AssumeRole to the arn of the role allowing Grafana access to AWS account B (the one I want to read Cloudwatch metrics from). The task already did have a ‘Task execution role’, which only has the permissions required for Grafana locally within AWS account A.
I have now added permissions from the Task role to the Task execution role as well, but there is no difference at all. Grafana still shows the same error indicating that id does not see any credentials: ‘send request failed caused by: Get “…/latest/meta-data/iam/security-credentials/”: dial tcp 169.254.169.254:80: connect: invalid argument’. According to Amazon’s documentation at Task IAM role - Amazon Elastic Container Service it should be looking for relative URI ‘/credential_provider_version/credentials?’, if I understand the documentation correctly.
And, by the way, Grafana does work and read Clodwatch data from account B if I just configure a user accout from AWS account B in Grafana’s datasource in account A…
What other information/configuration would you require to debug my problem?
Task role is:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Effect”: “Allow”,
“Action”: [
“sts:AssumeRole”
],
“Resource”: [
“arn:aws:iam::XXX-ID-account-B-XXX:role/Grafana-monitoring”
],
“Condition”: {
“ArnLike”: {
“aws:SourceArn”: “arn:aws:ecs:eu-central-1:XXX-ID-account-A-XXX:"
},
“StringEquals”: {
“aws:SourceAccount”: “XXX-ID-account-A-XXX”
}
}
}
]
}
…and the Trusted entities for that role are:
{
“Version”: “2012-10-17”,
“Statement”: [
{
“Sid”: “”,
“Effect”: “Allow”,
“Principal”: {
“Service”: “ecs-tasks.amazonaws.com”
},
“Action”: “sts:AssumeRole”,
“Condition”: {
“StringEquals”: {
“aws:SourceAccount”: “XXX-ID-account-A-XXX”
},
“ArnLike”: {
“aws:SourceArn”: "arn:aws:ecs:eu-central-1:XXX-ID-account-A-XXX:”
}
}
}
]
}
Thank you for any more hints!
I would do a standard debugging: increase log level and check/share a logs. I use cross account access from AWS ECS Fargate for ages (with older Grafana version) and I don’t have a problem.
Sorry for the delay - I haven’t had the time to pursue this in the meantime.
I now raised Grafana’s log level to debug, but that does not help in any way: there are a number of log messages when I try to save and test my datasource, but the only one relevant just give the same information I receive on the GUI: logger=tsdb.cloudwatch t=2023-05-25T09:59:45.365317486Z level=error msg=“error handling resource request” error=“error getting accounts for current user or role: ListSinks error: NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, .\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/\”: dial tcp 169.254.169.254:80: connect: invalid argument"
logger=tsdb.cloudwatch t=2023-05-25T09:59:45.365317486Z level=error msg=“error handling resource request” error=“error getting accounts for current user or role: ListSinks error: NoCredentialProviders: no valid providers in chain\ncaused by: EnvAccessKeyNotFound: failed to find credentials in the environment.\nSharedCredsLoad: failed to load profile, .\nEC2RoleRequestError: no EC2 instance role found\ncaused by: RequestError: send request failed\ncaused by: Get "http://169.254.169.254/latest/meta-data/iam/security-credentials/\”: dial tcp 169.254.169.254:80: connect: invalid argument".
To me this looks like Grafana possibly cannot make the http request to obtain meta-data from the EC2 instance my ECS task runs on?
