When I use loki query_range endpoint to query some data, it is noticed that the ‘total_bytes’ displaying in the log is growing all the time. However,I don’t understand which part of the data should the ‘total_bytes’ represents.
query_type=metric range_type=range length=24h0m0s step=24h0m0s duration=8.278945333s status=200 limit=100 returned_lines=0 throughput=137MB total_bytes=1.1GB queue_time=67.142µs
query_type=metric range_type=range length=32h2m4s step=24h0m0s duration=8.283767984s status=200 limit=100 returned_lines=0 throughput=773MB total_bytes=6.4GB queue_time=6.353596802s
Everything goes well in the first beginning after I deployed loki to K8s cluster. But after perhaps 1 month (?), the query request returns me error code 502
query_type=metric range_type=range length=48h0m0s step=24h0m0s duration=33.76475963s status=200 limit=100 returned_lines=0 throughput=775MB total_bytes=26GB queue_time=6.863727084s
the ‘total_bytes’ can be up to 26GB
In the meanwhile, Loki pod’s memory and CPU usage are both quite high
NAME CPU(cores) MEMORY(bytes)
loki-0 3784m 6627Mi
Some of the time, the memory usage of Loki can go up to 9GB
To fix the 502 issue temporary,I re-installed loki.Eveything works fine again.
Can I know is it something like a ‘cache’ issue ?
Anyone know this issue and how to prevent the 502 ???
Thank you very much