I don’t believe it needs UDP. We are running Loki with ECS as well, haven’t had problem.
Couple of suggestions if you aren’t already doing these:
- Separate your writer and readers. The easiest way to do this would be to setup one ECS cluster but two autoscaling groups. You can give them different
ECS_INSTANCE_ATTRIBUTES, and configure your ECS service to go to those instances withplacement_constraints.
The reason for doing this is primarily because of WAL requirements for the writers. They need dedicated persistent volumes for WAL, and ECS simply doesn’t have that functionality. The best workaround I could think of is to run writers as DAEMON so they can have unique bind mounts from the host.
- You’ll need service discovery for both writers and readers for ring membership, make sure to use
Arecord, don’t do SRV. CloudMap / Route53 is also limited to 8 service discovery record if you weren’t aware of this, so make sure you consider that when sizing your containers and make sure the scaling limit is 8.
You’ll also want to make sure your ECS services are using AWSVPC network mode because you need each writer and reader container to be individually discoverable, which means they each need unique IP to avoid port conflict. If you want to run more than a couple of containers on one host you’ll need to enable network interface trunking for your ECS hosts.
I had some additional discussions with someone else who’s also running Loki on ECS a while ago, there might be something in there that could be helpful: Loki 2.4.1 empty ring Code(500) error for "GET /loki/api/v1/labels" API on AWS ECS - #10 by tonyswumac