How can we set up Fail2ban to protect our dashboard?

Hello!

I have some dashboards open to internet and I would like to protect them, but I saw that Grafana logs does not take the IP of failed logins.

How can we protect our dashboard from a bruteforce attack?

This is an example of my logs located on: /var/log/grafana/grafana.log
(I rewrite the IPs and emails)

t=2019-10-15T09:11:04+0200 lvl=info msg=“Request Completed” logger=context userId=2 orgId=1 uname=user1 method=GET path=/api/datasources/proxy/1/query status=400 remote_addr=190.720.751.765 time_ms=20 size=77 referer=“https://myserver .com/graf/d/jOI1Ku7Wk/principal-fv?orgId=1&refresh=30s&fullscreen&edit&panelId=40”

t=2019-10-15T15:58:19+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/d/jOI1Ku7Wk/principal-fv status=302 remote_addr=190.720.751.765 time_ms=0 size=34 referer=
t=2019-10-15T15:58:43+0200 lvl=eror msg=“Invalid username or password” logger=context userId=0 orgId=0 uname= error=“Invalid Username or Password”

t=2019-10-15T15:58:43+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=190.720.751.765 time_ms=2 size=42 referer=https://myserver.com/graf/login

t=2019-10-15T15:58:48+0200 lvl=eror msg=“Invalid username or password” logger=context userId=0 orgId=0 uname= error=“Invalid Username or Password”

t=2019-10-15T15:58:48+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=190.720.751.765 time_ms=0 size=42 referer=https://myserver.com/graf/login

t=2019-10-15T15:58:55+0200 lvl=eror msg=“Invalid username or password” logger=context userId=0 orgId=0 uname= error=“Invalid Username or Password”

t=2019-10-15T15:58:55+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=190.720.751.765 time_ms=0 size=42 referer=https://myserver.com/graf/login

t=2019-10-15T18:45:58+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/d/jOI1Ku7Wk/principal status=302 remote_addr=190.720.751.765 time_ms=0 size=34 referer=

t=2019-10-15T18:46:08+0200 lvl=info msg=“Successful Login” logger=http.server User=user2@gmail .com

Grafana has login protection enabled by default (unless disabled).

think it will block logins for 5m or so after 5 failed attempts from the same IP and user

1 Like

Thanks you, it is great Grafana have an automatic protection, but I can not control it or log that it works.

That’s why I would like to know if is possible log the IP who fail at sign in.

Another option is to put an auth_proxy before Grafana and manage everything you want :wink:

1 Like

Thanks a lot, using auth_proxy could be a possible solution, but I prefer use the main login page of Grafana

I think i could open a request on GitHub to able log failed logins. It does not seem difficult

Did you end up opening a request? I also would like this, as I am trying to set up fail2ban for all of my websites.

No, I forgive that. Sorry. It is very important.

I will open it and quote you

EDIT: here is the request in GitHub: https://github.com/grafana/grafana/issues/21310

1 Like

This is now working in 6.7.0-beta1!!! Thank you @castillo92 for your efforts in getting this addressed!

For those interested, this is what I used for my failregex in fail2ban:

^ lvl=[a-zA-z]* msg=\"Invalid username or password\" (?:\S*=(?:\".*\"|\S*) )*remote_addr=<HOST>

I welcome any comments/critique on my regex as I am definitely not a regex expert.

*edit: changed regex to use non-capturing groups instead of capturing groups

1 Like

Thanks you very much @tkohhh

Today, three month later I setup the rules in fail2ban and it is working with a minor modification of your Regex:

My log files has the timestamp at the beggining of each line, so I needed to add .* after ^

Example of logs:

t=2020-04-29T21:41:36+0200 lvl=eror msg=“Invalid username or password” logger=context userId=0 orgId=0 uname= error=“Invalid Username or Password” remote_addr=88.9.13.3
t=2020-04-29T21:41:36+0200 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=POST path=/login status=401 remote_addr=88.9.13.3 time_ms=0 size=42 referer=https://myserver.com/graf/login

So the regex config file has to be: I only changed the beggining:

[Definition]
 
 failregex = ^.*lvl=[a-zA-z]* msg=\"Invalid username or password\" (?:\S*=(?:\".*\"|\S*) )*remote_addr=<HOST>
 
 ignoreregex =

Tried and works in 6.7.1 stable and, after the update, it still works so in 6.7.3 stable.

Glad to hear you got it working!

Are you sure it wasn’t working without the .* at the beginning? According to the fail2ban manual:

If the failregex is anchored with a leading ^, then the anchor refers to the start of the remainder of the line, after the timestamp and intervening whitespace.

Did you put a datepattern in your .conf file? I wonder if that makes a difference? Mine looks like this:

[Init]
datepattern = ^t=%%Y-%%m-%%dT%%H:%%M:%%S%%z

In any case, mine works without the leading .* and I think it’s generally a bad idea to have an unlimited match at the beginning of the failregex because it could potentially be abused with an injection-type attack.

Just to be clear, here is my whole grafana.conf. It works well for me.

[INCLUDES]
before = common.conf

[Definition]
failregex = ^ lvl=[a-zA-z]* msg=\"Invalid username or password\" (?:\S*=(?:\".*\"|\S*) )*remote_addr=<HOST>

ignoreregex =

[Init]
datepattern = ^t=%%Y-%%m-%%dT%%H:%%M:%%S%%z

Great, but would you be so kind and tell me how can I change this parameter in Grafana in docker?
I guess I have to add something in grafana.env?
Thank You!

The above syntax no longer works. The IP is listed on another line and isn’t being captured by fail2ban.

2023-09-28 07:39:33,017 fail2ban.filter [1]: WARNING [grafana_proxy] Please check a jail for a timing issue. Line with odd timestamp: logger=authn.service t=2023-09-28T08:39:32.757980995-06:00 level=warn msg=“Failed to authenticate request” client=auth.client.form error=“[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found”

Below is what I’m currently using, and I just confirmed that it’s still working (in Grafana 10.0). Grafana changed their log output format a while ago.

[Init]
datepattern = ^(?:[^=]+=[^ ]* )+t=%%Y-%%m-%%dT%%H:%%M:%%S.%%f(?:\d{0,6})%%z

[Definition]
failregex = msg="Invalid username or password" (?:[^=]+=[^.+]+ )+remote_addr=<ADDR>

Trying your configuration, however now I’m not getting anything in the fail2ban logs on a failed login attempt.

Did you restart fail2ban and confirm that it’s reading the grafana.log file in the initialization?

Yes I did. I even put my old configuration in and it would acknowledge it sees a failed login.
Also checked the grafana log and confirmed the failed login was being captured.

logger=authn.service t=2023-09-28T11:41:43.388752983-06:00 level=warn msg=“Failed to authenticate request” client=auth.client.form error=“[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found”

logger=context userId=0 orgId=0 uname= t=2023-09-28T11:41:43.389087901-06:00 level=info msg=“Bad request” error=“[password-auth.failed] failed to authenticate identity: [identity.not-found] no user fund: user not found” remote_addr=... traceID=

logger=context userId=0 orgId=0 uname= t=2023-09-28T11:41:43.392043647-06:00 level=info msg=“Request Completed” method=POST path=/login status=400 remote_addr=... time_ms=45 duration=45.078868ms size=107 referer=https://gr****/login handler=/login

What version of Grafana are you using? That log looks different from what I get. My log messages have this line:
msg="Invalid username or password"

where yours has:
msg="Bad request"

I’m on 10.1.2.

I can update later and see if I see the same thing, but in the mean time, based on the sample logs you sent, I think this should work for your failregex:
msg="Bad request" (?:[^=]+=[^.+]+ )+remote_addr=<ADDR>