AWS holds metrics for terminated instances for 15 months. I’d like to be able to query these using the CloudWatch data source. My use case is that we regularly cycle our instances and would like to see this old and new instances on our dashboards.
Presently, our dashboard looks like this after cycling the instances. The new ones are present but the old ones have been lost.
I’m generating the above graph using a variable containing a list of instance IDs. The query is:
ec2_instance_attribute(eu-west-1, InstanceId, { "tag:Name": [ "service"], "tag:Environment": [ "production"], "instance-state-name": ["running", "terminated"] })
I’m attempting to filter on the instance state for both running and terminated instances, but only the two running instances are being returned.
What do I need to do to get the metrics for my terminated instances to appear in my dashboard?
I know this is an old post but I was looking for the answer to this question today. I managed to do this using dimension_values to search in the actual metrics. The problem with that is that instance tags aren’t present on the metrics themselves. So what I did was added a custom dimension to the CloudWatch agent configuration. In my case it was Service, but you’ll probably want to add Environment and service to at least one of your metrics. Then you can add two variables to the dashboard configuration. For me this works:
Service:
dimension_values(default,CWAgent,mem_used_percent,Service)
Instance:
dimension_values(default,CWAgent,mem_used_percent,InstanceId,{"Service":["${Service}"]})`
Now you can find the instanceIds for any instance that’s reported the mem_used_percent metric to CloudWatch and use that as the dimension for other metrics.
Hope this makes sense. I used Grafana 8.2.2 so ymmv.
