I have a cloud based multi-VM setup with multiple docker containers of the same service.
On each VM, an nginx container acts as reverse proxy to load balance across the service containers and utilize the VM resources fully.
This is a node.js service, I have integrated Prometheus metrics in the application am emitting the metrics at container-ip:portXYZ/metrics. If I do a curl on that while sshed on the host, I am able to see the custom metrics I’m emitting.
Used the following packages in the application code to generate custom metrics
I also set up Grafana Agent (Linux Integration) on the VMs and am able to see the host level metrics in Grafana Cloud at the moment. I think the agent makes them available @ host:12345/metrics
Coming back to custom metrics, If query the api enpoint i.e api.org.com/metrics it returns metrics from any one of the container across the multiple VMs.
How to collate these custom metrics to per host level and push to Grafana cloud?
So if I select a VM I should be able to see the collated data across all the containers.