how often loki reads/and write from the blob storage, can someone provide some information on that, and is there a way to prevent loki from costing us that much. it costs us 6k USD and needs to fix it.
i guess these are the recommended configs
we got 10GB of files written to the blob storage over the last 4 months, and we do not have a clue why we are getting charged at this rate
It depends. If your concern is cost you aren’t concerned about the write, you are concerned about the total storage and read.
To reduce total storage you should set a comfortable retention policy, you can do this either from Loki compactor or S3 lifecycle policy, or both.
For read, there is not much you can do here. It depends on how often queries are executed against Loki. you can implement caching to reduce it some what, but whether that’s a saving depends on your volume of usage.
how often does Grafana try to flush chunks if something fails?. how many log lines does it produce in such situations? is it around a few hundred a second?