I need the ability to restrict the metrics and specific insights queries that my users can execute against an AWS Account.
I cannot use the security features of Grafana regarding dashboard configuration, as the API for Grafana allows the user to submit any query for a datasource they have access to, ie it is not restricted to those metric queries / insights queries configured by the administrator when setting up the dashboard.
So to get around this, I want to have an AWS HTTP Gateway, that is protected by IAM. The user configured in IAM only has API execute rights on my gateway, ie no rights to query cloudwatch directly and the lambda behind the gateway will inspect the request and only forward onto CloudWatch / Insights if it matches allowed values.
However, when I attempt to configure a CloudWatch datasource in Grafana, it generates the AWS signature v4 with the incorrect context.
AWS does not support any restriction on metrics. Once you are allowed access to getMetricData and getMetricStatistics it is a free for all for that authenticated user. Most metrics are not sensitive, but an example is if you enable DynamoDB insights, then the primary key values are published into metric data.
Addtionally for Cloudwatch log insights, the permissions of which do allow for restricting to specific log files, I need to interrogate log files that have sensitive information. My insights query accounts for this by only returning specific matches via filtering and regexâs.
So this is why I need the lambda. The lambda will have sufficient permissions to extract all the data, but in code, which will be controlled by a small number of privileges users, it will restrict which metrics and which insights queries are exposed via the API Gateway.
Also, as an example of what I am doing for insights.
LogGroup: /myapp/application-log
parse request /^(?[A-Z]+\s+/ensrv/[a-z/]*(?ns=[a-z0-9]+)?)/
| parse request /^(?POST - HTTP/1.1)$/
| stats max(request_time) as Max (s) by coalesce(ParsedRequest,EmptyRequest,â(not parsed)â) as CleanedRequest
| filter @message like /peak_memory/
| sort Max (s) desc
| limit 50
This is being executed against a log group whose raw data must not be exposed via the internet - even via authenticated IAM credentials.
My intention is that I would fake the describeLogGroups response, such that my lambda has these pre packaged queries,
Grafana config:
LogGroup : /myapp/peakemory
fields Max (s), CleanedRequest
sort Max (s)
This hits the lambda via API gateway, which executes the original query above using itâs internal role permissions to query the actual log group
Looking at the Grafana source code, I see that it just uses the AWS Go SDK. And the AWS SDK does not support overriding the service name when making a request unfortunately.
I will have to investigate a different method of auth.
I would say that needs to be implemented as another auth option in the CloudWatch datasource. You may enable sigv4 auth for https datasources, but then it wonât be a CloudWatch datasource anymore.
Yep, Unfortunately though it doesnât look like the AWS SDK supports an âoverride service nameâ option. Iâve raised a support request with AWS to confirm this.
