CloudWatch datasource and the Log Group according to a variable

Grafana
1

Hi, All,

The company I work for have been trying to achieve the following:

Imagine we have 3 different AWS accounts to which we deploy the exact same CloudFormation stack. These represent Dev, Test, and Prod environments for the same application.

There’s a Lambda Function in the stack, which prints logs to a CloudWatch Log Group called /aws/lambda/EventEmitter. The same log group exists in the 3 accounts with the exact same name.

The company want to configure 3 CloudWatch datasources in Grafana, each pointing to a different account. The team then want to create a dashboard where the 3 possible datasources are presented in a drop down list, so that the user can choose which environment they want to look at.

Visualizations in the dashboard would be querying the logs in /aws/lambda/EventEmitter, but the query would have to be executed against a different datasource every time the user selects a different one from the drop down menu.

I could find a way to present the datasources as values the user can choose from, as a variable in the dashboard. However, it looks like visualizations that query CloudWatch Log Groups hold on to more details about the log groups than just their names. When I select a different value for the variable in the drop down, the query against the new account fails with an authorization error. I can see in the error message that it is still trying to access the Log Group from the data source that was previously selected.

Is there a way to achieve this in Grafana?

I essence, I’d need to be able to select a different datasource from the drop down menu and then automatically adjust any visualization in the dashboard to query the Log Group through the newly selected data source.

2

Check these dashboards:

to see how you can switch cloudwatch datasources in dashboards.

3

Thank you. I checked this repository, and I could not find any example that queries CloudWatch Logs from a different CloudWatch datasource upon selection of a different datasource in the drop down list of a variable.

Instead, I found examples that query different CloudWatch Metrics, and that’s not the same as querying the contents of the log groups themselves.

I looked at this, for example, and it’s querying only metrics: AWS CloudWatch Logs | Grafana Labs

Same thing for these other examples provided in the repository:

  • AWS Cloudwatch Browser
  • AWS Cloudwatch Synthetics
  • AWS Cloudwatch Usage Metrics

I could not post the links to the examples above here because this forum limits the number of links a new user can post.

The problem seems to be the following: When I create a query over a Log Group, Grafana saves the entire ARN of the Log Group and the account number to the JSON representation of the dashboard. Example:

{
  "annotations": { ... },
  "description": "My dashboard",
  "editable": true,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 0,
  "id": 1300,
  "links": [],
  "panels": [
    ...
    {
      "datasource": {
        "type": "cloudwatch",
        "uid": "someUID"
      },
      "title": "Event publishing execution count",
      "fieldConfig": { ... },
      "gridPos": { ... },
      "id": 7,
      "options": { ... },
      "targets": [
        {
          "datasource": {
            "type": "cloudwatch",
            "uid": "someUID"
          },
          "dimensions": {},
          "expression": "fields @timestamp, @duration\n| filter @type = 'REPORT'\n| sort @timestamp desc\n| stats avg(@duration) as average, max(@duration) as maximum, min(@duration) as minimum by bin(1h)",
          "id": "",
          "label": "",
          "logGroups": [
            {
              "accountId": "000000000000",
              "arn": "arn:aws:logs:us-east-1:000000000000:log-group:/aws/lambda/event-publishing-lambda:*",
              "name": "/aws/lambda/event-publishing-lambda"
            }
          ],
          "matchExact": true,
          "metricEditorMode": 1,
          "metricName": "",
          "metricQueryType": 0,
          "namespace": "",
          "period": "",
          "queryMode": "Logs",
          "refId": "A",
          "region": "us-east-1",
          "sqlExpression": "",
          "statistic": "Average",
          "statsGroups": [
            "bin(1h)"
          ]
        }
      ],
      "transparent": true,
      "type": "timeseries"
    },
    ...
  ],
  "schemaVersion": 39,
  "tags": [],
  "templating": { ... },
  "time": { ... },
  "timepicker": {},
  "timezone": "browser",
  "title": "Event publishing dashboard",
  "uid": "someID",
  "version": 11,
  "weekStart": ""
}

Only the Log Group name is displayed in the UI, but the entire ARN and the account number are saved to the JSON. So far, I could not find a way to use a variable to dynamically query a different Log Group (with the exact same name) from a different account upon selection of a different data source from a variable.

I also did a search over all files in the repository, and I couldn’t find any JSON with the word logGroups. So, I take it as further evidence that there isn’t any example there that is doing what I am looking for.

Is it even possible? Or is it a limitation Grafana currently has?

1 Like
4

Correct, as I said that’s example for datasource switching - that’s one step in your case.
As you see you need also account ID. So try to create variable for account ID (just simple manual list) and use it in the arn - arn is not editable in the UI, so edit it in the panel json manually. If it works then improve account id variable - make it hidden with automatic lookup of account id based on datasource variable.