Very basic alert for elasticsearch filebeat logs config howto

[cartor79]

Hi all.

As newbie as it could sound, I couldnt find a post clarifying my doubt about how to set a Grafana’s alert rule that get trigguered when in the last 10 mins a patter in the ‘message’ field has been found.

Some of the logs I can see with a ES filebeat datasource look like this:

time="2023-12-17T17:27:35+01:00" level=info msg="COMPACT deleted 271 rows from 271 revisions in 3.190347ms - compacted to 102461/103461"

I’m trying to set an alert rule that get fired when the pattern ‘rows from’ is found at least twice along the last 10 minutes.

This is what I’ve tried:

But once I save and exit, I can see the rule in ‘normal status’ and never been fired.

Can you please tell what I’m missing.

Thank you very much and regards.

Carlos T

[jangaraj]

You are using last(), but that last value is mostly 0. I would use another function, e.g. max(), so condition will be true if there is at least one records in last 10 minutes.
+ you need also condition IS ABOVE 0, not IS ABOVE 1.

You have Preview button in the alert definition, so you can easily test it in the alert editor, how different condition works.

[cartor79]

Hi Jangaraj,

Thank you very much for your fast answer and help.

Indeed, your are right, using ‘last()’ does not make any sense
The thing is that I tried with max(), but I did some tests and the results weren’t those I was expecting.
I want to get an alert when the pattern is found more than a certain number of times along a period of time.
I think ‘max()’ will only work when the required number of patterns matched were logged exactly at the same time

Here is example in which I’m looking for the pattern “rows from” inside the ‘message’ field.

In the graph you can see how the pattern can be found every 5 minutes, but at 21:30, exactly at that precisely same moment, it happened twice. I checked it in ‘explore’ and in Kibana (Elastic GUI).

So if I’m setting max() above 1 along 10 minutes period, if will only work when the pattern is written two or more times at the same precisely moment.

I think I have to use sum() instead.

This is what I did:

I created four rules to match the pattern ‘rows from’ along the last 30 mins, and I called them:
rows from more than four, rows from more than five, … six… seven

This is how ‘rows from more than four’ looks like:

The other 3 rules check for values above, 5, 6 and 7 respectively.

If you see in the graph of the previous screenshot you can see how the pattern was found six times along last 30 mins.

Guess what rules were trigguered and with what alerting value

As 6 is above 4 or 5

Hope it makes sense and I’m missing nothing

Thank you for the help and regards.

Carlos T