Loki and clickhouse performance

Grafana Loki
1

clickhouse performance

We have a lot of applications and now we are collecting logs into clickhouse. With grafana query, we select the application to query based on project_name, app_name and in grafana we have a text input box where we can enter the keywords we want to query.
For example, I have selected project_name = “aaa”, app_name = “bbb” and entered a keyword “ccc” in the text input box. the above conditions will be combined into a sql to query in clickhouse: ``

select project_name, app_name, message,xxx from target_table where project_name = "aaa" and app_name = "bbb" and message like '%ccc%' and message_time >= $ _from and message_time <= $_to;
  • The number of log entries per app_name is about 300 million per day.
  • The timeframe of the query may be hours, days or a week.
  • You can order by asc/desc within the selected time range.
  • grafana also counts the number of eligible logs in a selected time range.

The response time of this query in clickhouse is quite fast. A query for a day’s worth of data (300 million+) will take a few seconds at most.

loki performance

In loki, however, a query for 30 million rows is very slow. If you look up the keyword, the loki service will out of memory. Also, I’ve found that count logs are the same, and loki is much, much slower than clickhouse.

question

Is loki’s performance bad if there is a keyword query?

Also, count is very slow.

2

How are you running your Loki cluster right now?

I’ve not use clickhouse before, so I can’t really comment on that. But for Loki most of the query performance comes from parallel processing and query splitting. If you have a big enough number of queries you can achieve pretty good query time.