AWS Cloudwatch using ARN in Grafana inside Kubernetes with uswitch/kiam

Grafana
1

Hello there, I deployed Grafana inside our k8s cluster with kind statefulset ,
here I need to add cloudwatch plugins using ARN as Auth Provider, using https://github.com/uswitch/kiam to communicate all our service with AWS IAM,

I already deep dive in Grafana AWS Cloudwatch Official Docs and https://github.com/uswitch/kiam/blob/master/docs/IAM.md,
and searching in grafana community here,
but no luck,

I already attached this to Trust Relationship in my grafana role too :

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/kiam-server"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

any one can help me to accomplished this?

noticable log :
in kiam-server

{"credentials.access.key":"XXXXXXXXXXXXXXX","credentials.expiration":"2019-12-16T06:24:19Z","credentials.role":"arn:aws:iam::XXXXXXXXXXXXXXX:role/grafana-dev-iam-role","generation.metadata":0,"level":"info","msg":"fetched credentials","pod.iam.role":"arn:aws:iam::XXXXXXXXXXXXXXX:role/grafana-dev-iam-role","pod.name":"grafana-0","pod.namespace":"istio-system","pod.status.ip":"","pod.status.phase":"Pending","resource.version":"45664633","time":"2019-12-16T06:06:05Z"}

in grafana pods :

t=2019-12-16T06:15:47+0000 lvl=eror msg="Metric request error" logger=context userId=1 orgId=1 uname=admin error="Failed to call ec2:getAwsConfig, AccessDenied: User: arn:aws:sts::XXXXXXXXXXXXXXX:assumed-role/grafana-dev-iam-role/kiam-kiam is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::XXXXXXXXXXXXXXX:role/grafana-dev-iam-role\n\tstatus code: 403, request id: 7ff33a3a-1fcb-11ea-b0f2-d3fa2c1d9a6b"

#aws #kiam

1 Like
2

It it not clear how did you configure all roles and why you need all of them: role/grafana-justice-dev-iam-role, role/grafana-dev-iam-role.

3

Had the same issue and I’m still investigating, maybe it is related to this open issue - https://github.com/grafana/grafana/issues/20473

However I have found a strage fix - you need to change credentials type from “arn” to “credentials” (Credentials file) leaving the path to the file empty (default)
After that change - everything works as expected.

Hope that helps,
Roman

4

Wow thank you so much for the hint! I will propose this to be added to the Grafana docs

5

Hi! Actually this fix is not required anymore, linked github issue is solved and all works as expected with assumeRoleArn and authtype: arn in grafana 7.1.1+