Cloudwatch Datasource - Proxy via AWS HTTP gateway for security, incorrect IAM signature generated

I need the ability to restrict the metrics and specific insights queries that my users can execute against an AWS Account.

I cannot use the security features of Grafana regarding dashboard configuration, as the API for Grafana allows the user to submit any query for a datasource they have access to, ie it is not restricted to those metric queries / insights queries configured by the administrator when setting up the dashboard.

So to get around this, I want to have an AWS HTTP Gateway, that is protected by IAM. The user configured in IAM only has API execute rights on my gateway, ie no rights to query cloudwatch directly and the lambda behind the gateway will inspect the request and only forward onto CloudWatch / Insights if it matches allowed values.

However, when I attempt to configure a CloudWatch datasource in Grafana, it generates the AWS signature v4 with the incorrect context.

It is sending the following authorization header

AWS4-HMAC-SHA256 Credential=AKIA2VRBQA32STVA5AVQ/20210331/ap-southeast-2/monitoring/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=51c119a7146bb99b05909d6b21d3fe21336c85bbb03b9dc77d7033e8fa9c34ec

But it needs to be

AWS4-HMAC-SHA256 Credential=AKIA2VRBQA32STVA5AVQ/20210331/ap-southeast-2/execute-api/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=51c119a7146bb99b05909d6b21d3fe21336c85bbb03b9dc77d7033e8fa9c34ec

Is this something that could be made configurable in Grafana?

Why you don’t configure those limitations on used IAM role/user level?

AWS does not support any restriction on metrics. Once you are allowed access to getMetricData and getMetricStatistics it is a free for all for that authenticated user. Most metrics are not sensitive, but an example is if you enable DynamoDB insights, then the primary key values are published into metric data.

Addtionally for Cloudwatch log insights, the permissions of which do allow for restricting to specific log files, I need to interrogate log files that have sensitive information. My insights query accounts for this by only returning specific matches via filtering and regex’s.

So this is why I need the lambda. The lambda will have sufficient permissions to extract all the data, but in code, which will be controlled by a small number of privileges users, it will restrict which metrics and which insights queries are exposed via the API Gateway.

Did you try to conditions keys? The example for namespace limitation: Using Condition Keys to Limit Access to CloudWatch Namespaces - Amazon CloudWatch

Unfortunately the namespace condition key only operates on PutMetricData. GetMetricData has no such limitiations

Also, as an example of what I am doing for insights.

LogGroup: /myapp/application-log

parse request /^(?[A-Z]+\s+/ensrv/[a-z/]*(?ns=[a-z0-9]+)?)/
| parse request /^(?POST - HTTP/1.1)$/
| stats max(request_time) as Max (s) by coalesce(ParsedRequest,EmptyRequest,’(not parsed)’) as CleanedRequest
| filter @message like /peak_memory/
| sort Max (s) desc
| limit 50

This is being executed against a log group whose raw data must not be exposed via the internet - even via authenticated IAM credentials.

My intention is that I would fake the describeLogGroups response, such that my lambda has these pre packaged queries,

Grafana config:

LogGroup : /myapp/peakemory

fields Max (s), CleanedRequest
sort Max (s)

This hits the lambda via API gateway, which executes the original query above using it’s internal role permissions to query the actual log group

Looking at the Grafana source code, I see that it just uses the AWS Go SDK. And the AWS SDK does not support overriding the service name when making a request unfortunately.

I will have to investigate a different method of auth.

1 Like

I would say that needs to be implemented as another auth option in the CloudWatch datasource. You may enable sigv4 auth for https datasources, but then it won’t be a CloudWatch datasource anymore.

Yep, Unfortunately though it doesn’t look like the AWS SDK supports an “override service name” option. I’ve raised a support request with AWS to confirm this.