Unauthorized Error: Grafana HA with Aurora MySQL

  • What Grafana version and what operating system are you using?

I’m running grafana/grafana-oss:10.4.1 on Kubernetes 1.26, AMD Nodes.

  • What are you trying to achieve?

I have installed Grafana using the OAuth Generic authentication method. Now, I would like to scale Grafana horizontally to achieve HA using AWS Aurora MySQL(engine=8.0.mysql_aurora.3.06.0) database.

  • How are you trying to achieve it?

I have being followed this Guide Set up Grafana for high availability | Grafana documentation

  • What happened?

When I run multiple instances, after a couple of seconds of being logged in, everytime I click a button, it throws me an “Unauthorized” notification on my browser and then I get logged out.

After login again, it throws the same error on every action I make on the UI.

  • What did you expect to happen?

I expect to work properly without being logged out.

  • Can you copy/paste the configuration(s) that you are having problems with?
    Sure!

    ################################## Database ##############################
    [database]
    type = mysql
    host = grafana.cluster.aurora.mysql.host:3306
    user = root
    password = password

    #################################### Generic OAuth ##########################
    [auth.generic_oauth]
    enabled = true
    name = NAME
    allow_sign_up = true
    client_id = grafana-dev
    client_secret = secret
    scopes = openid grafana-dev atp u:email,read:org
    auth_url = https://auth_url/authorization
    tls_skip_verify_insecure=true
    role_attribute_path = “‘Admin’”
    skip_org_role_sync = true
    token_url = http://token_url/token
    api_url = https://api_url/userinfo
    redirect_uri = https://redirect_uri/login/generic_oauth
    use_refresh_token = false
    ;team_ids =
    ;allowed_organizations =

    [sso_settings]
    reload_interval = 0
    configurable_providers = generic_oauth

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

I found this line on one Grafana instance:

“Failed to authenticate request” client=auth.client.session error=“user token not found”

  • Did you follow any online instructions? If so, what is the URL?

I have being followed this Guide Set up Grafana for high availability | Grafana documentation

Findings:
I tried to reproduce the issue using Aurora PostgreSQL(version 14.7) and I couldn’t reproduce it.
Apparently it happens only with Aurora MySQL.

Let me know If I can provide more details or if there is another place to post this issue.

Thank you in advance.