We are starting to use autoscale with EC2 to create/destroy nodes based on time or random metric X in our application. Hostnames will be based on Amazon instanceid (ie www-prod-aejfth5849349412.x), so we will have dozens of dead/deregistered nodes running around the graphs. Is there a good strategy to use for cleaning up dead entries in the graphs on this kind of scale?
I am using Icinga2 -> Graphite -> Grafana, and the node will be deleted in the Icinga2 API during autoscale termination. For Graphite, probably a script that deletes .wsp files based on -mtime ?
Does anyone have a great cleanup strategy for use with autoscale or ephemeral nodes?