Guidance on ensuring availability of LGTM services during EKS cluster upgrade

[thomasplarsson]

Hello,

Apologies for this rather broad question, feel free to suggest I narrow it down.

In short, I’m asking for documentation or advice on how to ensure continuous availability of LGTM services during an EKS cluster upgrade.

Longer.

We have an LGTM stack deployed on AWS EKS using the standard helm charts. We need to upgrade the kubernetes version of EKS and would prefer if all services were available during the upgrade.

Per service we are using these charts/deployment modes:

• loki - SimpleScalable
• mimir - mimir-distributed
• tempo - tempo-distributed
• grafana - standard chart with replicas=2

As a preparatory step we ran the eksup tool which identifies potential risks for an EKS upgrade. When we ran this we got a fairly large number of warnings, such as that the loki-gateway is only running with a single replica (this is just one example warning). The explanation for this warning (k8s002) says:

“Multiple replicas, along with the use of PodDisruptionBudget, are REQUIRED to ensure high availability during the upgrade.”

I can understand that if a k8s upgrade is performed and there are such services that only run with 1 replica, you will likely have a period of outage? But I’m fairly new to the LGTM stack so perhaps some components are not as important as others.

I have tried to look around the documentation for information on how to handle this type of upgrade but I haven’t found anything.

Are there any guides on this topic somewhere or does someone have any other advice to share?

If you are operating a similar stack, how are you handling these types of upgrades?

Best Regards

[infofcc3]

Welcome @thomasplarsson to the community
The warnings from eksup are generally valid. During an EKS upgrade, nodes are drained and pods are evicted. Any critical LGTM component running with a single replica can experience downtime while Kubernetes reschedules it.

To minimize disruption, I would recommend:

→ Running at least 2 replicas for critical components (for example loki-gateway, query frontends, distributors, Grafana, etc.)
→ Configuring PodDisruptionBudgets (PDBs)
→ Using pod anti-affinity or topology spread constraints so replicas are placed on different nodes

I tested the HA behavior locally with a deployment using 2 replicas and a PDB

When one pod was deleted, Kubernetes immediately created a replacement while another replica remained available, which is the same principle used during node draining and EKS upgrades.

A good validation step before the actual upgrade is to manually drain a node in a non-production environment and verify that Grafana, Loki, Mimir, and Tempo remain available throughout the process.