Cannot setup AWS CloudWatch Prometheus metrics in Dashboard

[zillag]

• What Grafana version and what operating system are you using?
  9.4

• What are you trying to achieve?
  I’m trying to setup Prometheus metrics from my ECS containers that are in CloudWatch on Grafana board

• How are you trying to achieve it?

1. I set up an ECS microservice with a Java agent to collect JMX metrics accessible through port 9404
2. I set up an ECS Prometheus “collector” service to scrape the metrics on port 9404 and send them to CloudWatch
3. I see the following in CloudWatch

{
    "CloudWatchMetrics": [
        {
            "Metrics": [
                {
                    "Name": "jvm_memory_pool_bytes_used"
                }
            ],
            "Dimensions": [
                [
                    "ClusterName",
                    "TaskDefinitionFamily",
                    "pool"
                ]
            ],
            "Namespace": "ECS/ContainerInsights/Prometheus"
        }
    ],
    "ClusterName": "feature",
    "ECS_PROMETHEUS_EXPORTER_PORT": "9404",
    "Java_EMF_Metrics": "true",
    "LaunchType": "FARGATE",
    "StartedBy": "ecs-svc/3763850937600431306",
    "TaskDefinitionFamily": "my-service-feature",
    "TaskGroup": "service:my-service",
    "TaskRevision": "25",
    "Timestamp": "1708563763792",
    "Version": "0",
    "container_name": "my-service-feature",
    "instance": "10.23.x.x:9404",
    "job": "cwagent-ecs-file-sd-config",
    "jvm_memory_pool_bytes_committed": 85983232,
    "jvm_memory_pool_bytes_init": 0,
    "jvm_memory_pool_bytes_max": -1,
    "jvm_memory_pool_bytes_used": 82473672,
    "pool": "Metaspace",
    "prom_metric_type": "gauge"
}

4. I set up an AWS Grafana workspace, and choosing CloudWatch us-east-1 as a data source
5. When I try to setup a Dashboard, I don’t see the “ECS/ContainerInsights/Prometheus” namespace in the “Namespace” dropdown list.

• What happened?
  When I try to setup a Dashboard, I don’t see the “ECS/ContainerInsights/Prometheus” namespace in the “Namespace” dropdown list.

• What did you expect to happen?
  I expect to see the “ECS/ContainerInsights/Prometheus” in the Namespace dropdown list. See attached.
  

  

• Can you copy/paste the configuration(s) that you are having problems with?
  I don’t think I have config issues.

• Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
  No.

• Did you follow any online instructions? If so, what is the URL?
  I started here Grafana dashboards overview | Grafana documentation