How to setup AWS ECS Monitoring

Grafana Configuration
1

Hi all,

We’ve recently started investigating moving to Grafana Cloud, and one of the things we’re trying to figure out is how to move our AWS ECS clusters with prometheus metrics, tracing, and logs and ensure there’s a simple way we can use the Grafana agent to forward all those things to Grafana cloud.

The blog post here suggests running the agent as a sidecar on our cluster, but doesn’t have any details, simply a “(Full documentation on this is coming soon!)” tag.

This forum post has a similar question, but no answers.

This discussion also asks about the same topic, but also has no input.

Does anyone know of an example that actually shows best practise in how to configure ECS to export to Grafana Cloud?

2

That’s complex topic.

Simple solution is to use existing CloudWatch metrics - grafana-aws-cloudwatch-dashboards/aws-ecs at master · monitoringartist/grafana-aws-cloudwatch-dashboards · GitHub

I wouldn’t use Grafana agent for complex solution, but I would go with ADOT, so you will have AWS supported solution. ADOT is just OpenTelemetry collector, so you can configure ADOT to send metrics and traces to Grafana Cloud Opentelemetry endpoint.

Logs: subscribe CloudWatch logs and send them via Lambda to Loki Grafana Cloud endpoint (implement it in your favourite language, e.g. python).

3

Ah, thanks for all the links, there’s a lot here that’s completely new to me so I’ll spend a few days trying to make heads and tails of it all. Is there a reason you state you wouldn’t use Grafana Agent for complex solutions? From what I was reading in the blog post I got the impression that Grafana Agent was where development was happening and the recommended way to gather these resources. Does it have some significant pitfalls you’ve encountered?

4

Generally, if you read something from Grafana Labs then they will try to use maximum from their stack (Grafana, Tempo, Loki, Grafana Agent, …). If you read from AWS then they will try to use also max from their stack (CloudWatch, X-Ray, ADOT, …). That’s normal. But in this case Grafana Agent is light vendor lock in. Again it’s normal. It can be good option if you have Grafana support = you can request support directly from Grafana support for your use case.
Everything has pros/cons. Yes, you may think that ADOT solution needs a days - absolutely. But that can be case also for Grafana agent, when you don’t have any example how to deploy it.
Anyway, this is personal opinion. Only, you know your case and what you can/can’t accept, so you will be able to find right solution for your case.

5

Sorry for the necromancy, but I feel it’s better to continue the thread.

@jangaraj regarding the logs, what do you think about the FireLens sidecar ?
It makes me think it would make the Terraform configuration clearer by keeping things in one place instead of having an extra Lambda. (working on an implementation but there’s reading and reading :slight_smile: )

6

I would say it depends:

  • cost - will it be cheaper/expensive than Lambda setup
  • knowledge - do you know how to use that
  • standards - is it widely supported infrastructure in the compay
  • support - will need to be supported solution
  • protocols - will it work with supported protocols in your observability stack
    …
1 Like
7

The only question, in our case, is about the cost (knowledge is limited in any direction, we’re just starting with IaC so it will be ‘standard’, we use Grafana Cloud so there’s some support )

I’ll try to follow this (although seems dated) Loki tutorial: How to send logs from Amazon's ECS to Loki | Grafana Labs

1 Like