Loki stuck failing to flush on kubernetes cluster

"caller":"flush.go:144", "err":"failed to flush chunks: store put chunk: googleapi: Error 400: The maximum object length is 1024 characters, but got a name with 1325 characters:

I have three resources in my kubernetes namespace for loki.
read pods (statefulset), write pods (statefulset) and a gateway (ClusterIP internal service). I’ve killed the entire namespace and brought it back up and still see these logs on the write pods?

Also after deep investigation these logs are old - I had my agent attach new labels and the new label came through for all other logs except this log which continues to fail to flush.
I don’t have access to a prometheus dashboard or the LogCLI unless it’s a last resort.

using this helm chart → loki 4.8.0 · grafana/grafana

Any ideas?