Grafana Cloud Provider Authentication Error

Grafana Authentication
1

I’m using terraform to create resources for my Grafana Cloud instance but I’m getting an error that I can’t dig deeper into.

I’m trying to configure the AWS Observability Provider by creating a token that has the necessary permissions and instantiating a provider with the necessary configuration. When it comes time to create the AWS account resource by submitting a POST to the cloud API it fails and I can’t understand why. I am able to create the AWS account manually which confirms that the role and everything else I’m doing is valid.

Here is my configuration

provider "grafana" {
  alias = "cloud"
  // terraform cloud access policy
  cloud_access_policy_token = var.cloud_access_policy_token
}

data "grafana_cloud_stack" "main" {
  provider = grafana.cloud

  slug = "xxx"
}

data "grafana_cloud_organization" "current" {
  provider = grafana.cloud

  slug = "xxx"
}

resource "grafana_cloud_access_policy" "aws" {
  provider = grafana.cloud

  region       = data.grafana_cloud_stack.main.region_slug
  name         = "aws-cloudwatch-policy"
  display_name = "AWS Cloudwatch Policy"

  scopes = [
    "integration-management:read",
    "integration-management:write",
    "stacks:read"
  ]

  realm {
    type       = "org"
    identifier = data.grafana_cloud_organization.current.id
  }
}

resource "grafana_cloud_access_policy_token" "aws" {
  provider = grafana.cloud

  region           = data.grafana_cloud_stack.main.region_slug
  access_policy_id = grafana_cloud_access_policy.aws.policy_id
  name             = "aws-cloudwatch-policy-token"
  display_name     = "AWS Cloudwatch Policy Token"
}

provider "grafana" {
  alias = "aws"

  cloud_provider_url          = format("https://cloud-provider-api-%s.grafana.net", data.grafana_cloud_stack.main.cluster_slug)
  cloud_provider_access_token = grafana_cloud_access_policy_token.aws.token
}

resource "grafana_cloud_provider_aws_account" "main" {
  provider = grafana.aws

  stack_id = data.grafana_cloud_stack.main.id
  role_arn = "xxx"

  regions = [
    "us-west-1",
    "us-west-2"
  ]

}

resource "grafana_cloud_provider_aws_cloudwatch_scrape_job" "main" {
  provider = grafana.aws

  stack_id                = data.grafana_cloud_stack.main.id
  name                    = "aws-resource-metadata-scraper"
  aws_account_resource_id = grafana_cloud_provider_aws_account.main.resource_id

  service {
    name = "AWS/CertificateManager"

    metric {
      name       = "DaysToExpiry"
      statistics = ["Average"]
    }

    scrape_interval_seconds = 86400
  }
}

And here is the subsequent error I get during resource creation

grafana_cloud_provider_aws_account.main: Creating...
grafana_cloud_provider_aws_account.main: Still creating... [10s elapsed]
╷
│ Error: Failed to create AWS Account
│
│   with grafana_cloud_provider_aws_account.main,
│   on main.tf line 582, in resource "grafana_cloud_provider_aws_account" "main":
│  582: resource "grafana_cloud_provider_aws_account" "main" {
│
│ failed to create AWS account: failed to do request: Post "https://cloud-provider-api-prod-us-central-3.grafana.net/api/v2/stacks/xxx/aws/accounts": POST
│ https://cloud-provider-api-prod-us-central-3.grafana.net/api/v2/stacks/xxx/aws/accounts giving up after 4 attempt(s)
2

Is there any particular reason why you need to configure cloud_provider_url explicitly?
Does TF has network connectivity to that endpoint https://cloud-provider-api-prod-us-central-3.grafana.net/api/v2/stacks/xxx/aws/accounts? Can you run tf in debug mode to understand what is the error - timeout, error code, …? What is duration of create aws account when you are using UI - is it more than 10 secs?

3

  1. It’s a requirement for configuring the AWS account as stated in this document here.

  2. It should have network connectivity, I don’t see any reason it would be blocked.

  3. Running tf in debug mode shows that the error code being returned by the POST is a 500.

  4. It’s faster than 10s, I believe the 10s is indicating that it’s tried to POST 4 times in 10s and has received a 500 response each time.

4

OK, use latest provider version and if it still returns 5xx then contact Grafana support.