Grafana Server Admin Role needed when login with Keycloak

  • What Grafana version and what operating system are you using?
    Grafana 10.0.3 - Running as POD in mickrok8s cluster

  • What are you trying to achieve?
    Want to login to Grafana dashboard with keycloak authentication, which is working fine. But icons like Explore, Server Admin, Configuration, Server Admin etc is missing in the Grafana UI. What roles should I add in keycloak to get “Full admin privileges” in the grafana UI for a used logged in through keycloak

  • How are you trying to achieve it?
    Integrated keycloak with Grafana but as mentioned, not getting all the required privileges in Grafana UI.

  • What happened?
    All Icons as mentioned above is not available, I have added admin, editor and viewer roles in grafana.ini and respective roles is added in keycloak. In Grafana UI, few icons are coming, but not all. I expect all the privileges.

  • What did you expect to happen?
    WOuld like to have all the “super admin” priviledges in grafana UI

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

  • Did you follow any online instructions? If so, what is the URL?

See: Configure generic OAuth2 authentication | Grafana documentation :

role_attribute_path - JMESPath expression to use for Grafana role lookup. Grafana will first evaluate the expression using the OAuth2 ID token. If no role is found, the expression will be evaluated using the user information obtained from the UserInfo endpoint. The result of the evaluation should be a valid Grafana role (Viewer, Editor, Admin or GrafanaAdmin). For more information on user role mapping, refer to Configure role mapping.

So map user role to GrafanaAdmin (not just Admin) + other config options mentioned in the linked in the doc.

Thanks for this Jaganraj. I will try to setup this configuration and will let you know.
Thank you

Grafana version is 10.0.3
Keycloak version is 21.1.2

This is the setting I provided in grafana.ini
role_attribute_path = contains(roles[], ‘grafanaadmin’) && ‘GrafanaAdmin’ || contains(roles[], ‘admin’) && ‘Admin’ || contains(roles[*], ‘editor’) && ‘Editor’ || ‘Viewer’
allow_assign_grafana_admin = true

I have created Role called GrafanaAdmin in Keycloak and assigned the role to the user.

I am able to login to Grafana dashboard with keycloak credentials, but even after having the “GrafanaAdmin” Role I am not able to see below ICONs / options
Explore
Configuration
Server Admin.

What is available for me is “Dashboard” and “Alerting” Only

Please prove that this config is valid and it matches your roles in access/id token or userinfo result.

My Requirement is to provide admin Or grafanaadmin (admin is also fine) role for a user logged in through keycloak.

Below is my grafana.ini
[auth]
disable_login_form = true
enable_login_token = true
oauth_auto_login = true
[auth.generic_oauth]
name = Keycloak-OAuth
enabled = true
allow_sign_up = true
client_id = grafana
client_secret = hlsS2EhwXR7uq0L0Ttg8HEQ0hAkt0bFr
scopes = openid profile email offline_access roles
email_attribute_path = email
login_attribute_path = username
name_attribute_path = full_name
auth_url = xxxxx.com
token_url = xxxxx.com
api_url = xxxxx.com
role_attribute_path = contains(roles[], ‘admin’) && ‘Admin’ || contains(roles[], ‘editor’) && ‘Editor’ || ‘Viewer’
ssl_ca_cert = /etc/ssl/certs/ca.crt
tls_skip_verify_insecure = true

My Requirment is to provide admin Or grafanaadmin (admin is also fine) role for a user logged in through keycloak.

I used the above configuration (grafana.ini) and used the below link to configure keycloak (keycloak version is 21.1.2)

My Image for grafana is → Image: grafana/grafana-oss:10.0.3.

I always get the “Viewer” role only for the user logged in with keycloak even if I gave admin “roles” in keycloak for the user.
(In Grafana UI, went to profile, it shows → Organizations → Name (Main Org.) → Role (Viewer)

Keycloak User Info
{
“sub”: “99dc20b8-ade1-4bd2-98e5-90ec16380e9e”,
“email_verified”: true,
“realm_access”: {
“roles”: [
“offline_access”,
“uma_authorization”,
“default-roles-platform”
]
},
“name”: “Dileep AP”,
“preferred_username”: “dileep”,
“given_name”: “Dileep”,
“family_name”: “AP”,
“email”: “dileep@example.com
}


Container Logs says → msg=“No valid role found. Skipping role sync. In Grafana 10, this will result in the user being assigned the default role and overriding manual assignment. If role sync is not desired, set skip_org_role_sync for your provider to true”


Additional Info : I tried below 2 things also.

  1. Provide grafanaamdin role, but still got “Viewer” role only
  2. As per this link Configure Grafana | Grafana documentation
    Set below options and tried and i still got veiwer role only for the user autheticated through keycloak
    [users]
    auto_assign_org = true
    auto_assign_org_role = Admin

Full Container log is attached below.

baa@k8sdbtest3:~$ kubectl logs -f loki-grafana-59d95f997f-dsdj8 -c grafana -n monitoring
logger=settings t=2023-08-20T16:33:27.8952882Z level=info msg=“Starting Grafana” version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
logger=settings t=2023-08-20T16:33:27.895448Z level=warn msg=“[Deprecated] The oauth_auto_login configuration setting is deprecated. Please use auto_login inside auth provider section instead.”
logger=settings t=2023-08-20T16:33:27.8955552Z level=warn msg=“"sentry" frontend logging provider is deprecated and will be removed in the next major version. Use "grafana" provider instead.”
logger=settings t=2023-08-20T16:33:27.8955683Z level=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-08-20T16:33:27.895574Z level=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2023-08-20T16:33:27.8955789Z level=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana/”
logger=settings t=2023-08-20T16:33:27.895584Z level=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2023-08-20T16:33:27.8955887Z level=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2023-08-20T16:33:27.8955978Z level=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2023-08-20T16:33:27.8956029Z level=info msg=“Config overridden from command line” arg=“default.log.mode=console”
logger=settings t=2023-08-20T16:33:27.8956117Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_DATA=/var/lib/grafana/”
logger=settings t=2023-08-20T16:33:27.8956166Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_LOGS=/var/log/grafana”
logger=settings t=2023-08-20T16:33:27.8956238Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PLUGINS=/var/lib/grafana/plugins”
logger=settings t=2023-08-20T16:33:27.8956285Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PROVISIONING=/etc/grafana/provisioning”
logger=settings t=2023-08-20T16:33:27.8956362Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_USER=admin”
logger=settings t=2023-08-20T16:33:27.8956418Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_PASSWORD=*********”
logger=settings t=2023-08-20T16:33:27.8956491Z level=info msg=Target target=[all]
logger=settings t=2023-08-20T16:33:27.8956624Z level=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2023-08-20T16:33:27.8956679Z level=info msg=“Path Data” path=/var/lib/grafana/
logger=settings t=2023-08-20T16:33:27.8956724Z level=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2023-08-20T16:33:27.8956772Z level=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2023-08-20T16:33:27.895685Z level=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2023-08-20T16:33:27.8956896Z level=info msg=“App mode production”
logger=sqlstore t=2023-08-20T16:33:27.895869Z level=info msg=“Connecting to DB” dbtype=sqlite3
logger=sqlstore t=2023-08-20T16:33:27.8958848Z level=warn msg=“SQLite database file has broader permissions than it should” path=/var/lib/grafana/grafana.db mode=-rw-rw---- expected=-rw-r-----
logger=migrator t=2023-08-20T16:33:27.9129329Z level=info msg=“Starting DB migrations”
logger=migrator t=2023-08-20T16:33:27.9205383Z level=info msg=“migrations completed” performed=0 skipped=485 duration=861.3µs
logger=secrets t=2023-08-20T16:33:27.9211138Z level=info msg=“Envelope encryption state” enabled=true currentprovider=secretKey.v1
logger=local.finder t=2023-08-20T16:33:27.9770438Z level=warn msg=“Skipping finding plugins as directory does not exist” path=/usr/share/grafana/plugins-bundled
logger=query_data t=2023-08-20T16:33:27.97993Z level=info msg=“Query Service initialization”
logger=live.push_http t=2023-08-20T16:33:27.9831954Z level=info msg=“Live Push Gateway initialization”
logger=ngalert t=2023-08-20T16:33:27.9874193Z level=warn msg=“Unexpected number of rows updating alert configuration history” rows=0 org=1 hash=not-yet-calculated
logger=infra.usagestats.collector t=2023-08-20T16:33:29.6102611Z level=info msg=“registering usage stat providers” usageStatsProvidersLen=2
logger=provisioning.alerting t=2023-08-20T16:33:29.6105876Z level=info msg=“starting to provision alerting”
logger=provisioning.alerting t=2023-08-20T16:33:29.6106096Z level=info msg=“finished to provision alerting”
logger=modules t=2023-08-20T16:33:29.6108102Z level=warn msg=“No modules registered…”
logger=ngalert.state.manager t=2023-08-20T16:33:29.6109547Z level=info msg=“Warming state cache for startup”
logger=grafanaStorageLogger t=2023-08-20T16:33:29.6112222Z level=info msg=“storage starting”
logger=http.server t=2023-08-20T16:33:29.6140677Z level=info msg=“HTTP Server Listen” address=[::]:3000 protocol=https subUrl=/grafana socket=
logger=ngalert.state.manager t=2023-08-20T16:33:29.6260966Z level=info msg=“State cache has been initialized” states=0 duration=15.1385ms
logger=ticker t=2023-08-20T16:33:29.6262187Z level=info msg=starting first_tick=2023-08-20T16:33:30Z
logger=ngalert.multiorg.alertmanager t=2023-08-20T16:33:29.626244Z level=info msg=“starting MultiOrg Alertmanager”
logger=grafana.update.checker t=2023-08-20T16:33:29.9358367Z level=info msg=“Update check succeeded” duration=324.5597ms
logger=plugins.update.checker t=2023-08-20T16:33:30.1141036Z level=info msg=“Update check succeeded” duration=502.5771ms
logger=context t=2023-08-20T16:36:54.5280486Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:54.5287147Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=192.168.254.1 time_ms=2 duration=2.0512ms size=37 referer=https://k8sdbtest3.xxxxxxxx.com/mdap/platform/dashboard handler=/
logger=context t=2023-08-20T16:36:54.6534716Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:54.6554236Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:54.6554764Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=192.168.254.1 time_ms=2 duration=2.2893ms size=64 referer=https://k8sdbtest3.xxxxxxxx.com/mdap/platform/dashboard handler=/login
logger=context t=2023-08-20T16:36:54.7799391Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:54.7802476Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=192.168.254.1 time_ms=0 duration=533.5µs size=308 referer=https://k8sdbtest3.xxxxxxxx.com/mdap/platform/dashboard handler=/login/:name
logger=context t=2023-08-20T16:36:55.0501154Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=oauth t=2023-08-20T16:36:55.050424Z level=info msg=“state check” queryState=b8f6cbcb4e5abeabe3f84621e7327ab895e42645493e49e7f606fd8a9de70038 cookieState=b8f6cbcb4e5abeabe3f84621e7327ab895e42645493e49e7f606fd8a9de70038
logger=oauth.generic_oauth t=2023-08-20T16:36:55.1667947Z level=warn msg=“No valid role found. Skipping role sync. In Grafana 10, this will result in the user being assigned the default role and overriding manual assignment. If role sync is not desired, set skip_org_role_sync for your provider to true”
logger=http.server t=2023-08-20T16:36:55.2732961Z level=info msg=“Successful Login” User=dileep@example.com
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:55.2733769Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=192.168.254.1 time_ms=223 duration=223.5667ms size=32 referer= handler=/login/:name
logger=context userId=2 orgId=1 uname=dileep@example.com t=2023-08-20T16:36:56.396371Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=192.168.254.1 time_ms=0 duration=766.5µs size=0 referer= handler=/api/live/ws
logger=context t=2023-08-20T16:36:56.639269Z level=warn msg=“failed to look up session from cookie” error=“context canceled”
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:56.6393189Z level=warn msg= error=“context canceled” traceID=
logger=context userId=0 orgId=0 uname= t=2023-08-20T16:36:56.6393628Z level=info msg=“Request Completed” method=GET path=/avatar/7180247b3df28ebda398f5bb4d0c6b73 status=403 remote_addr=192.168.254.1 time_ms=0 duration=142.9µs size=15 referer=“https://k8sdbtest3.xxxxxxxx.com/grafana/?orgId=1


Your userinfo output doesn’t show that you have assigned admin role, so Grafana doesn’t grant you admin role. You have wrong Keycloak configuration - it is not a Grafana problem.

Thank You. I followed below documentation to setup keycloak for grafana

I created a client grafana → In that created roles called “admin & editor”. Assigned this roles to the user (dileep).

Is there any detailed documentation for setting up the keyclaok for grafana.
Thanks

That doc is Grafana doc, so it maintains how to set up Grafana. Keycloak is just used IPD in that example. Keycloak has many config options, so check Keycloak doc how to configure it. Generally, you must have details about user roles in access/id token/user info and only then can Grafana use them.

Thanks Jangaraj for the Support. I was able to make it work with some changes in the keycloak role mapperes. Also, I need to take the roles from ID token instead of user info.

With the below document, i was able to set it up correctly.

Thanks for the support from your side.

But not sure how can i validate if GrafanaAdmin privilege is granted for the user as I can see only “Admin” privilege granted if i validate in the “Profile” section of grafana UI.
The user logged in with keycloak IDP

role_attribute_path = contains(resource_access.grafana.roles[*], 'admin') && 'Admin' || contains(resource_access.grafana.roles[*], 'grafanaadmin') && 'GrafanaAdmin' || 'Viewer'
allow_assign_grafana_admin = true

If you have access to server admin, then you have role GrafanaAdmin.

1 Like

I don’t get the ICON with the shield “Server Admin”. I have created & mapped 2 roles in keycloak.
admin and grafanaadmin
{
“exp”: 1692879441,
“iat”: 1692879141,
“auth_time”: 0,
“jti”: “07df5038-ee8e-40fc-a6e1-6676b03163bb”,
“iss”: “https://k8sdbtest1.xxxxxxxx.com/auth/realms/platform”,
“aud”: “grafana”,
“sub”: “a2cfbf83-f26a-4e19-bfcd-b7b014cbaebe”,
“typ”: “ID”,
“azp”: “grafana”,
“session_state”: “14f48506-0fa8-41b1-88ff-9c9efbf5c2ed”,
“acr”: “1”,
“sid”: “14f48506-0fa8-41b1-88ff-9c9efbf5c2ed”,
“resource_access”: {
“realm-management”: {
“roles”: [
“view-realm”,
“manage-realm”,
“view-events”,
“manage-users”,
“manage-authorization”,
“query-groups”,
“query-users”
]
},
“xxxx-xxxx”: {
“roles”: [
“xxxx-management”,
“xxxx-management”,
“xxxx-xxxx”,
“xxxx-xxxx”
]
},
“grafana”: {
“roles”: [
“admin”,
“grafanaadmin”
]
},
“account”: {
“roles”: [
“manage-account”,
“manage-account-links”,
“view-profile”
]
}
},
“email_verified”: true,
“realm_access”: {
“roles”: [
“offline_access”,
“uma_authorization”,
“default-roles-platform”
]
},
“name”: “user”,
“preferred_username”: “user”,
“given_name”: “user”,
“family_name”: “AP”,
“email”: “user@xxxxxxxx.com
}

grafana.ini
[auth]
disable_login_form = true
enable_login_token = true
oauth_auto_login = true
signout_redirect_url = https://k8sdbtest1.xxxxxxxx.com/auth/realms/platform/protocol/openid-connect/logout?redirect_uri=https%3A%2F%2Fk8sdbtest1.xxxxxxx.com/xxxx/platform%2Flogin
[auth.generic_oauth]
name = Keycloak-OAuth
enabled = true
allow_sign_up = true
client_id = grafana
client_secret = zg9OUAq4BJYsbRiUOTw1CJLqyVlvwk37
scopes = openid profile email offline_access roles
email_attribute_path = email
login_attribute_path = username
name_attribute_path = full_name
auth_url = xxxxx.com
token_url = xxxxx.com
api_url = https://k8sdbtest1.xxxxxxxx.com/realms/platform/protocol/openid-connect/userinfo
role_attribute_path = contains(resource_access.grafana.roles[], ‘admin’) && ‘Admin’ || contains(resource_access.grafana.roles[], ‘grafanaadmin’) && ‘GrafanaAdmin’ || ‘Viewer’
allow_assign_grafana_admin = true
ssl_ca_cert = /etc/ssl/certs/ca.crt
tls_skip_verify_insecure = true

Container Logs
aaa@k8sdbtest1:~$ kubectl logs -f loki-grafana-5b4ffd74d6-6vb52 -c grafana -n monitoring
logger=settings t=2023-08-24T08:21:14.2354056Z level=info msg=“Starting Grafana” version=10.0.3 commit=eb8dd72637 branch=HEAD compiled=2023-07-25T17:55:59Z
logger=settings t=2023-08-24T08:21:14.2355281Z level=warn msg=“[Deprecated] The oauth_auto_login configuration setting is deprecated. Please use auto_login inside auth provider section instead.”
logger=settings t=2023-08-24T08:21:14.2356597Z level=warn msg=“"sentry" frontend logging provider is deprecated and will be removed in the next major version. Use "grafana" provider instead.”
logger=settings t=2023-08-24T08:21:14.235678Z level=info msg=“Config loaded from” file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2023-08-24T08:21:14.2356834Z level=info msg=“Config loaded from” file=/etc/grafana/grafana.ini
logger=settings t=2023-08-24T08:21:14.235688Z level=info msg=“Config overridden from command line” arg=“default.paths.data=/var/lib/grafana/”
logger=settings t=2023-08-24T08:21:14.2356924Z level=info msg=“Config overridden from command line” arg=“default.paths.logs=/var/log/grafana”
logger=settings t=2023-08-24T08:21:14.2356972Z level=info msg=“Config overridden from command line” arg=“default.paths.plugins=/var/lib/grafana/plugins”
logger=settings t=2023-08-24T08:21:14.2357058Z level=info msg=“Config overridden from command line” arg=“default.paths.provisioning=/etc/grafana/provisioning”
logger=settings t=2023-08-24T08:21:14.2357111Z level=info msg=“Config overridden from command line” arg=“default.log.mode=console”
logger=settings t=2023-08-24T08:21:14.23572Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_DATA=/var/lib/grafana/”
logger=settings t=2023-08-24T08:21:14.2357253Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_LOGS=/var/log/grafana”
logger=settings t=2023-08-24T08:21:14.2357347Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PLUGINS=/var/lib/grafana/plugins”
logger=settings t=2023-08-24T08:21:14.2357395Z level=info msg=“Config overridden from Environment variable” var=“GF_PATHS_PROVISIONING=/etc/grafana/provisioning”
logger=settings t=2023-08-24T08:21:14.2357479Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_USER=admin”
logger=settings t=2023-08-24T08:21:14.2357886Z level=info msg=“Config overridden from Environment variable” var=“GF_SECURITY_ADMIN_PASSWORD=*********”
logger=settings t=2023-08-24T08:21:14.2357922Z level=info msg=Target target=[all]
logger=settings t=2023-08-24T08:21:14.2357998Z level=info msg=“Path Home” path=/usr/share/grafana
logger=settings t=2023-08-24T08:21:14.2358028Z level=info msg=“Path Data” path=/var/lib/grafana/
logger=settings t=2023-08-24T08:21:14.2358058Z level=info msg=“Path Logs” path=/var/log/grafana
logger=settings t=2023-08-24T08:21:14.2358086Z level=info msg=“Path Plugins” path=/var/lib/grafana/plugins
logger=settings t=2023-08-24T08:21:14.2358115Z level=info msg=“Path Provisioning” path=/etc/grafana/provisioning
logger=settings t=2023-08-24T08:21:14.2358146Z level=info msg=“App mode production”
logger=sqlstore t=2023-08-24T08:21:14.2360996Z level=info msg=“Connecting to DB” dbtype=sqlite3
logger=sqlstore t=2023-08-24T08:21:14.2361615Z level=warn msg=“SQLite database file has broader permissions than it should” path=/var/lib/grafana/grafana.db mode=-rw-rw---- expected=-rw-r-----
logger=migrator t=2023-08-24T08:21:14.2515488Z level=info msg=“Starting DB migrations”
logger=migrator t=2023-08-24T08:21:14.2557801Z level=info msg=“Executing migration” id=“Update is_service_account column to nullable”
logger=migrator t=2023-08-24T08:21:14.27335Z level=info msg=“Executing migration” id=“Add isPublic for dashboard”
logger=migrator t=2023-08-24T08:21:14.2786316Z level=info msg=“Executing migration” id=“set service account foreign key to nil if 0”
logger=migrator t=2023-08-24T08:21:14.2821287Z level=info msg=“Executing migration” id=“Add last_used_at to api_key table”
logger=migrator t=2023-08-24T08:21:14.2865698Z level=info msg=“Executing migration” id=“Add is_revoked column to api_key table”
logger=migrator t=2023-08-24T08:21:14.2918003Z level=info msg=“Executing migration” id=“Add column preferences.json_data”
logger=migrator t=2023-08-24T08:21:14.2965768Z level=info msg=“Executing migration” id=“alter preferences.json_data to mediumtext v1”
logger=migrator t=2023-08-24T08:21:14.3005755Z level=info msg=“Executing migration” id=“Add preferences index org_id”
logger=migrator t=2023-08-24T08:21:14.307728Z level=info msg=“Executing migration” id=“Add preferences index user_id”
logger=migrator t=2023-08-24T08:21:14.3124436Z level=info msg=“Executing migration” id=“Increase tags column to length 4096”
logger=migrator t=2023-08-24T08:21:14.3170104Z level=info msg=“Executing migration” id=“Add OAuth ID token to user_auth”
logger=migrator t=2023-08-24T08:21:14.3220973Z level=info msg=“Executing migration” id=“alter table short_url alter column created_by type to bigint”
logger=migrator t=2023-08-24T08:21:14.326702Z level=info msg=“Executing migration” id=“add current_reason column related to current_state”
logger=migrator t=2023-08-24T08:21:14.3350002Z level=info msg=“Executing migration” id=“add rule_group_idx column to alert_rule”
logger=migrator t=2023-08-24T08:21:14.3429798Z level=info msg=“Executing migration” id=“add is_paused column to alert_rule table”
logger=migrator t=2023-08-24T08:21:14.3480522Z level=info msg=“Executing migration” id=“fix is_paused column for alert_rule table”
logger=migrator t=2023-08-24T08:21:14.3518931Z level=info msg=“Executing migration” id=“add rule_group_idx column to alert_rule_version”
logger=migrator t=2023-08-24T08:21:14.3564977Z level=info msg=“Executing migration” id=“add is_paused column to alert_rule_versions table”
logger=migrator t=2023-08-24T08:21:14.3615099Z level=info msg=“Executing migration” id=“fix is_paused column for alert_rule_version table”
logger=migrator t=2023-08-24T08:21:14.3655102Z level=info msg=“Executing migration” id=“add configuration_hash column to alert_configuration”
logger=migrator t=2023-08-24T08:21:14.3703835Z level=info msg=“Executing migration” id=“add column send_alerts_to in ngalert_configuration”
logger=migrator t=2023-08-24T08:21:14.3753875Z level=info msg=“Executing migration” id=“create provenance_type table”
logger=migrator t=2023-08-24T08:21:14.3804553Z level=info msg=“Executing migration” id=“add index to uniquify (record_key, record_type, org_id) columns”
logger=migrator t=2023-08-24T08:21:14.3853716Z level=info msg=“Executing migration” id=“create alert_image table”
logger=migrator t=2023-08-24T08:21:14.3906803Z level=info msg=“Executing migration” id=“add unique index on token to alert_image table”
logger=migrator t=2023-08-24T08:21:14.3954161Z level=info msg=“Executing migration” id=“support longer URLs in alert_image table”
logger=migrator t=2023-08-24T08:21:14.3997177Z level=info msg=“Executing migration” id=create_alert_configuration_history_table
logger=migrator t=2023-08-24T08:21:14.4052954Z level=info msg=“Executing migration” id=“drop non-unique orgID index on alert_configuration”
logger=migrator t=2023-08-24T08:21:14.4098034Z level=info msg=“Executing migration” id=“drop unique orgID index on alert_configuration if exists”
logger=migrator t=2023-08-24T08:21:14.4099465Z level=warn msg=“Skipping migration: Already executed, but not recorded in migration log” id=“drop unique orgID index on alert_configuration if exists”
logger=migrator t=2023-08-24T08:21:14.4145525Z level=info msg=“Executing migration” id=“extract alertmanager configuration history to separate table”
logger=migrator t=2023-08-24T08:21:14.4188868Z level=info msg=“Executing migration” id=“add unique index on orgID to alert_configuration”
logger=migrator t=2023-08-24T08:21:14.429255Z level=info msg=“Executing migration” id=“add last_applied column to alert_configuration_history”
logger=migrator t=2023-08-24T08:21:14.4348237Z level=info msg=“Executing migration” id=“increase max description length to 2048”
logger=migrator t=2023-08-24T08:21:14.4426147Z level=info msg=“Executing migration” id=“alter library_element model to mediumtext”
logger=migrator t=2023-08-24T08:21:14.447119Z level=info msg=“Executing migration” id=“create secrets table”
logger=migrator t=2023-08-24T08:21:14.4530859Z level=info msg=“Executing migration” id=“rename data_keys name column to id”
logger=migrator t=2023-08-24T08:21:14.4638063Z level=info msg=“Executing migration” id=“add name column into data_keys”
logger=migrator t=2023-08-24T08:21:14.4684333Z level=info msg=“Executing migration” id=“copy data_keys id column values into name”
logger=migrator t=2023-08-24T08:21:14.4723239Z level=info msg=“Executing migration” id=“rename data_keys name column to label”
logger=migrator t=2023-08-24T08:21:14.4832539Z level=info msg=“Executing migration” id=“rename data_keys id column back to name”
logger=migrator t=2023-08-24T08:21:14.493627Z level=info msg=“Executing migration” id=“add column hidden to role table”
logger=migrator t=2023-08-24T08:21:14.4983764Z level=info msg=“Executing migration” id=“create query_history table v1”
logger=migrator t=2023-08-24T08:21:14.503051Z level=info msg=“Executing migration” id=“add index query_history.org_id-created_by-datasource_uid”
logger=migrator t=2023-08-24T08:21:14.5073178Z level=info msg=“Executing migration” id=“alter table query_history alter column created_by type to bigint”
logger=migrator t=2023-08-24T08:21:14.5117701Z level=info msg=“Executing migration” id=“rbac disabled migrator”
logger=migrator t=2023-08-24T08:21:14.5162041Z level=info msg=“Executing migration” id=“teams permissions migration”
logger=migrator t=2023-08-24T08:21:14.5206772Z level=info msg=“Executing migration” id=“dashboard permissions”
logger=migrator t=2023-08-24T08:21:14.5319806Z level=info msg=“Executing migration” id=“dashboard permissions uid scopes”
logger=migrator t=2023-08-24T08:21:14.5625629Z level=info msg=“Executing migration” id=“drop managed folder create actions”
logger=migrator t=2023-08-24T08:21:14.5659416Z level=info msg=“Executing migration” id=“alerting notification permissions”
logger=migrator t=2023-08-24T08:21:14.570663Z level=info msg=“Executing migration” id=“create query_history_star table v1”
logger=migrator t=2023-08-24T08:21:14.5755975Z level=info msg=“Executing migration” id=“add index query_history.user_id-query_uid”
logger=migrator t=2023-08-24T08:21:14.5805295Z level=info msg=“Executing migration” id=“add column org_id in query_history_star”
logger=migrator t=2023-08-24T08:21:14.5856715Z level=info msg=“Executing migration” id=“alter table query_history_star_mig column user_id type to bigint”
logger=migrator t=2023-08-24T08:21:14.5900304Z level=info msg=“Executing migration” id=“create correlation table v1”
logger=migrator t=2023-08-24T08:21:14.5958771Z level=info msg=“Executing migration” id=“add index correlations.uid”
logger=migrator t=2023-08-24T08:21:14.6006976Z level=info msg=“Executing migration” id=“add index correlations.source_uid”
logger=migrator t=2023-08-24T08:21:14.6047897Z level=info msg=“Executing migration” id=“add correlation config column”
logger=migrator t=2023-08-24T08:21:14.6097814Z level=info msg=“Executing migration” id=“create entity_events table”
logger=migrator t=2023-08-24T08:21:14.6141876Z level=info msg=“Executing migration” id=“create dashboard public config v1”
logger=migrator t=2023-08-24T08:21:14.6193102Z level=info msg=“Executing migration” id=“drop index UQE_dashboard_public_config_uid - v1”
logger=migrator t=2023-08-24T08:21:14.619413Z level=warn msg=“Skipping migration: Already executed, but not recorded in migration log” id=“drop index UQE_dashboard_public_config_uid - v1”
logger=migrator t=2023-08-24T08:21:14.6246775Z level=info msg=“Executing migration” id=“drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1”
logger=migrator t=2023-08-24T08:21:14.6248406Z level=warn msg=“Skipping migration: Already executed, but not recorded in migration log” id=“drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1”
logger=migrator t=2023-08-24T08:21:14.6296444Z level=info msg=“Executing migration” id=“Drop old dashboard public config table”
logger=migrator t=2023-08-24T08:21:14.6343906Z level=info msg=“Executing migration” id=“recreate dashboard public config v1”
logger=migrator t=2023-08-24T08:21:14.6381773Z level=info msg=“Executing migration” id=“create index UQE_dashboard_public_config_uid - v1”
logger=migrator t=2023-08-24T08:21:14.6436178Z level=info msg=“Executing migration” id=“create index IDX_dashboard_public_config_org_id_dashboard_uid - v1”
logger=migrator t=2023-08-24T08:21:14.647883Z level=info msg=“Executing migration” id=“drop index UQE_dashboard_public_config_uid - v2”
logger=migrator t=2023-08-24T08:21:14.6516571Z level=info msg=“Executing migration” id=“drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2”
logger=migrator t=2023-08-24T08:21:14.6557608Z level=info msg=“Executing migration” id=“Drop public config table”
logger=migrator t=2023-08-24T08:21:14.6611944Z level=info msg=“Executing migration” id=“Recreate dashboard public config v2”
logger=migrator t=2023-08-24T08:21:14.6653904Z level=info msg=“Executing migration” id=“create index UQE_dashboard_public_config_uid - v2”
logger=migrator t=2023-08-24T08:21:14.6697944Z level=info msg=“Executing migration” id=“create index IDX_dashboard_public_config_org_id_dashboard_uid - v2”
logger=migrator t=2023-08-24T08:21:14.6739836Z level=info msg=“Executing migration” id=“create index UQE_dashboard_public_config_access_token - v2”
logger=migrator t=2023-08-24T08:21:14.6785708Z level=info msg=“Executing migration” id=“Rename table dashboard_public_config to dashboard_public - v2”
logger=migrator t=2023-08-24T08:21:14.6870854Z level=info msg=“Executing migration” id=“add annotations_enabled column”
logger=migrator t=2023-08-24T08:21:14.6921422Z level=info msg=“Executing migration” id=“add time_selection_enabled column”
logger=migrator t=2023-08-24T08:21:14.6978709Z level=info msg=“Executing migration” id=“delete orphaned public dashboards”
logger=migrator t=2023-08-24T08:21:14.7022316Z level=info msg=“Executing migration” id=“add share column”
logger=migrator t=2023-08-24T08:21:14.7071116Z level=info msg=“Executing migration” id=“backfill empty share column fields with default of public”
logger=migrator t=2023-08-24T08:21:14.7118137Z level=info msg=“Executing migration” id=“create default alerting folders”
logger=migrator t=2023-08-24T08:21:14.7124569Z level=info msg=“created the default folder for alerting” org_id=1 folder_name=“General Alerting” folder_uid=dfc6973d-d3e9-4471-a2de-f90f34a642f8
logger=migrator t=2023-08-24T08:21:14.7233676Z level=info msg=“Executing migration” id=“create file table”
logger=migrator t=2023-08-24T08:21:14.7277192Z level=info msg=“Executing migration” id=“file table idx: path natural pk”
logger=migrator t=2023-08-24T08:21:14.7335288Z level=info msg=“Executing migration” id=“file table idx: parent_folder_path_hash fast folder retrieval”
logger=migrator t=2023-08-24T08:21:14.7385373Z level=info msg=“Executing migration” id=“create file_meta table”
logger=migrator t=2023-08-24T08:21:14.7438638Z level=info msg=“Executing migration” id=“file table idx: path key”
logger=migrator t=2023-08-24T08:21:14.751031Z level=info msg=“Executing migration” id=“set path collation in file table”
logger=migrator t=2023-08-24T08:21:14.7559787Z level=info msg=“Executing migration” id=“managed permissions migration”
logger=migrator t=2023-08-24T08:21:14.7647165Z level=info msg=“Executing migration” id=“managed folder permissions alert actions migration”
logger=migrator t=2023-08-24T08:21:14.7708458Z level=info msg=“Executing migration” id=“RBAC action name migrator”
logger=migrator t=2023-08-24T08:21:14.7761888Z level=info msg=“Executing migration” id=“Add UID column to playlist”
logger=migrator t=2023-08-24T08:21:14.7815649Z level=info msg=“Executing migration” id=“Update uid column values in playlist”
logger=migrator t=2023-08-24T08:21:14.7899023Z level=info msg=“Executing migration” id=“Add index for uid in playlist”
logger=migrator t=2023-08-24T08:21:14.7991434Z level=info msg=“Executing migration” id=“update group index for alert rules”
logger=migrator t=2023-08-24T08:21:14.807111Z level=info msg=“Executing migration” id=“managed folder permissions alert actions repeated migration”
logger=migrator t=2023-08-24T08:21:14.8116353Z level=info msg=“Executing migration” id=“admin only folder/dashboard permission”
logger=migrator t=2023-08-24T08:21:14.8164899Z level=info msg=“Executing migration” id=“add action column to seed_assignment”
logger=migrator t=2023-08-24T08:21:14.8296967Z level=info msg=“Executing migration” id=“add scope column to seed_assignment”
logger=migrator t=2023-08-24T08:21:14.8401227Z level=info msg=“Executing migration” id=“remove unique index builtin_role_role_name before nullable update”
logger=migrator t=2023-08-24T08:21:14.8504889Z level=info msg=“Executing migration” id=“update seed_assignment role_name column to nullable”
logger=migrator t=2023-08-24T08:21:14.8719215Z level=info msg=“Executing migration” id=“add unique index builtin_role_name back”
logger=migrator t=2023-08-24T08:21:14.8773729Z level=info msg=“Executing migration” id=“add unique index builtin_role_action_scope”
logger=migrator t=2023-08-24T08:21:14.8835934Z level=info msg=“Executing migration” id=“add primary key to seed_assigment”
logger=migrator t=2023-08-24T08:21:14.8927682Z level=info msg=“Executing migration” id=“managed folder permissions alert actions repeated fixed migration”
logger=migrator t=2023-08-24T08:21:14.8999937Z level=info msg=“Executing migration” id=“migrate external alertmanagers to datsourcse”
logger=migrator t=2023-08-24T08:21:14.9044097Z level=info msg=“Executing migration” id=“create folder table”
logger=migrator t=2023-08-24T08:21:14.9089659Z level=info msg=“Executing migration” id=“Add index for parent_uid”
logger=migrator t=2023-08-24T08:21:14.9134129Z level=info msg=“Executing migration” id=“Add unique index for folder.uid and folder.org_id”
logger=migrator t=2023-08-24T08:21:14.9186656Z level=info msg=“Executing migration” id=“Update folder title length”
logger=migrator t=2023-08-24T08:21:14.9239333Z level=info msg=“Executing migration” id=“Add unique index for folder.title and folder.parent_uid”
logger=migrator t=2023-08-24T08:21:14.9308405Z level=info msg=“migrations completed” performed=109 skipped=376 duration=675.1092ms
logger=secrets t=2023-08-24T08:21:14.9313033Z level=info msg=“Envelope encryption state” enabled=true currentprovider=secretKey.v1
logger=local.finder t=2023-08-24T08:21:14.9807508Z level=warn msg=“Skipping finding plugins as directory does not exist” path=/usr/share/grafana/plugins-bundled
logger=query_data t=2023-08-24T08:21:14.9834996Z level=info msg=“Query Service initialization”
logger=live.push_http t=2023-08-24T08:21:14.9924105Z level=info msg=“Live Push Gateway initialization”
logger=ngalert t=2023-08-24T08:21:14.9976446Z level=warn msg=“Unexpected number of rows updating alert configuration history” rows=0 org=1 hash=not-yet-calculated
logger=infra.usagestats.collector t=2023-08-24T08:21:16.3928286Z level=info msg=“registering usage stat providers” usageStatsProvidersLen=2
logger=provisioning.alerting t=2023-08-24T08:21:16.410878Z level=info msg=“starting to provision alerting”
logger=provisioning.alerting t=2023-08-24T08:21:16.4109097Z level=info msg=“finished to provision alerting”
logger=modules t=2023-08-24T08:21:16.4110866Z level=warn msg=“No modules registered…”
logger=grafanaStorageLogger t=2023-08-24T08:21:16.4124639Z level=info msg=“storage starting”
logger=ngalert.state.manager t=2023-08-24T08:21:16.4154597Z level=info msg=“Warming state cache for startup”
logger=http.server t=2023-08-24T08:21:16.4223106Z level=info msg=“HTTP Server Listen” address=[::]:3000 protocol=https subUrl=/grafana socket=
logger=ngalert.state.manager t=2023-08-24T08:21:16.4286811Z level=info msg=“State cache has been initialized” states=0 duration=13.2205ms
logger=ticker t=2023-08-24T08:21:16.4287445Z level=info msg=starting first_tick=2023-08-24T08:21:20Z
logger=ngalert.multiorg.alertmanager t=2023-08-24T08:21:16.428753Z level=info msg=“starting MultiOrg Alertmanager”
logger=sqlstore.transactions t=2023-08-24T08:21:16.4590796Z level=info msg=“Database locked, sleeping then retrying” error=“database is locked” retry=0 code=“database is locked”
logger=grafana.update.checker t=2023-08-24T08:21:16.7804425Z level=info msg=“Update check succeeded” duration=368.9857ms
logger=plugins.update.checker t=2023-08-24T08:21:16.9517997Z level=info msg=“Update check succeeded” duration=539.1394ms
logger=ngalert t=2023-08-24T08:22:16.4313948Z level=warn msg=“Unexpected number of rows updating alert configuration history” rows=0 org=1 hash=not-yet-calculated
logger=context t=2023-08-24T08:25:23.2302235Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:25:23.230505Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=556.5µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context t=2023-08-24T08:25:23.3772148Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:25:23.378207Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:25:23.3782468Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=1 duration=1.232ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context t=2023-08-24T08:25:23.5263311Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:25:23.5266384Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=505.9µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=context t=2023-08-24T08:25:23.8302462Z level=warn msg=“failed to look up session from cookie” error=“user token not found”
logger=oauth t=2023-08-24T08:25:23.8304688Z level=info msg=“state check” queryState=f55962632591ff9c8564545792e7f66a136223707348489615e38133f3738b3d cookieState=f55962632591ff9c8564545792e7f66a136223707348489615e38133f3738b3d
logger=http.server t=2023-08-24T08:25:23.9468535Z level=info msg=“Successful Login” User=user@xxxxxxxx.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:25:23.9469233Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=116 duration=116.8995ms size=32 referer= handler=/login/:name
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:25:33.8414127Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=656.1µs size=0 referer= handler=/api/live/ws
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:25:54.1302639Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=72 duration=72.9098ms size=183 referer=https://k8sdbtest1.xxxxxxxx.com/grafana/profile handler=/logout
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:26:41.7312695Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=449.6µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:26:41.8876035Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:26:41.8876558Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=1 duration=1.2556ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:26:42.0361921Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=737.5µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=oauth t=2023-08-24T08:26:43.521827Z level=info msg=“state check” queryState=10a80296197386db6b0067fa37d9f6386f8ae0a89eac888bf71f8b71d3d685f4 cookieState=10a80296197386db6b0067fa37d9f6386f8ae0a89eac888bf71f8b71d3d685f4
logger=http.server t=2023-08-24T08:26:43.6437475Z level=info msg=“Successful Login” User=user@xxxxxxxx.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:26:43.6438153Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=122 duration=122.2609ms size=32 referer= handler=/login/:name
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:26:54.7075551Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=785.5µs size=0 referer= handler=/api/live/ws
logger=cleanup t=2023-08-24T08:31:16.4239545Z level=info msg=“Completed cleanup jobs” duration=9.0664ms
logger=grafana.update.checker t=2023-08-24T08:31:17.1228274Z level=info msg=“Update check succeeded” duration=341.8263ms
logger=plugins.update.checker t=2023-08-24T08:31:17.4777191Z level=info msg=“Update check succeeded” duration=525.6352ms
logger=cleanup t=2023-08-24T08:41:16.4280156Z level=info msg=“Completed cleanup jobs” duration=12.9945ms
logger=grafana.update.checker t=2023-08-24T08:41:17.1371151Z level=info msg=“Update check succeeded” duration=356.1927ms
logger=plugins.update.checker t=2023-08-24T08:41:17.5091323Z level=info msg=“Update check succeeded” duration=556.3069ms
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:44:09.6924486Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=665.6µs size=0 referer= handler=/api/live/ws
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:44:12.7484845Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=66 duration=66.3606ms size=183 referer=“https://k8sdbtest1.xxxxxxxx.com/grafana/?orgId=1” handler=/logout
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:49:50.0662944Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=295.9µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:49:50.2474087Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:49:50.2474608Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=2 duration=2.1057ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:49:50.3809989Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=332.3µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=oauth t=2023-08-24T08:49:50.6491839Z level=info msg=“state check” queryState=580de25e9a5b60fdfbd5328d3b20f0ac7e70d00925c0f09b129712cfb47e594f cookieState=580de25e9a5b60fdfbd5328d3b20f0ac7e70d00925c0f09b129712cfb47e594f
logger=http.server t=2023-08-24T08:49:50.759391Z level=info msg=“Successful Login” User=user@xxxxxxxx.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T08:49:50.7594615Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=110 duration=110.5744ms size=32 referer= handler=/login/:name
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T08:49:59.9553186Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=1 duration=1.2148ms size=0 referer= handler=/api/live/ws
logger=cleanup t=2023-08-24T08:51:16.4253718Z level=info msg=“Completed cleanup jobs” duration=10.2215ms
logger=grafana.update.checker t=2023-08-24T08:51:17.0951374Z level=info msg=“Update check succeeded” duration=313.6369ms
logger=plugins.update.checker t=2023-08-24T08:51:17.436693Z level=info msg=“Update check succeeded” duration=484.3751ms
logger=cleanup t=2023-08-24T09:01:16.424259Z level=info msg=“Completed cleanup jobs” duration=9.0676ms
logger=grafana.update.checker t=2023-08-24T09:01:17.1844775Z level=info msg=“Update check succeeded” duration=403.2429ms
logger=plugins.update.checker t=2023-08-24T09:01:17.5399711Z level=info msg=“Update check succeeded” duration=587.6022ms
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T09:07:19.4107932Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=65 duration=65.0089ms size=183 referer=“https://k8sdbtest1.xxxxxxxx.com/grafana/?orgId=1” handler=/logout
logger=cleanup t=2023-08-24T09:11:16.4193158Z level=info msg=“Completed cleanup jobs” duration=3.9163ms
logger=grafana.update.checker t=2023-08-24T09:11:17.2284746Z level=info msg=“Update check succeeded” duration=447.64ms
logger=plugins.update.checker t=2023-08-24T09:11:17.5509892Z level=info msg=“Update check succeeded” duration=598.8477ms
logger=cleanup t=2023-08-24T09:21:16.4194689Z level=info msg=“Completed cleanup jobs” duration=4.0409ms
logger=grafana.update.checker t=2023-08-24T09:21:17.1499951Z level=info msg=“Update check succeeded” duration=369.05ms
logger=plugins.update.checker t=2023-08-24T09:21:17.4724565Z level=info msg=“Update check succeeded” duration=520.1469ms
logger=cleanup t=2023-08-24T09:31:16.4195644Z level=info msg=“Completed cleanup jobs” duration=3.973ms
logger=grafana.update.checker t=2023-08-24T09:31:17.2586291Z level=info msg=“Update check succeeded” duration=477.9502ms
logger=plugins.update.checker t=2023-08-24T09:31:17.4704557Z level=info msg=“Update check succeeded” duration=517.6298ms
logger=cleanup t=2023-08-24T09:41:16.4192677Z level=info msg=“Completed cleanup jobs” duration=3.8698ms
logger=grafana.update.checker t=2023-08-24T09:41:17.1442149Z level=info msg=“Update check succeeded” duration=363.4637ms
logger=plugins.update.checker t=2023-08-24T09:41:17.4870043Z level=info msg=“Update check succeeded” duration=534.7784ms
logger=cleanup t=2023-08-24T09:51:16.4242209Z level=info msg=“Completed cleanup jobs” duration=9.3357ms
logger=grafana.update.checker t=2023-08-24T09:51:17.1224147Z level=info msg=“Update check succeeded” duration=341.345ms
logger=plugins.update.checker t=2023-08-24T09:51:17.5846175Z level=info msg=“Update check succeeded” duration=631.7558ms
logger=cleanup t=2023-08-24T10:01:16.4244315Z level=info msg=“Completed cleanup jobs” duration=9.3009ms
logger=grafana.update.checker t=2023-08-24T10:01:17.0870551Z level=info msg=“Update check succeeded” duration=305.6341ms
logger=plugins.update.checker t=2023-08-24T10:01:17.523348Z level=info msg=“Update check succeeded” duration=570.9458ms
logger=cleanup t=2023-08-24T10:11:16.4243968Z level=info msg=“Completed cleanup jobs” duration=9.1531ms
logger=grafana.update.checker t=2023-08-24T10:11:17.1668737Z level=info msg=“Update check succeeded” duration=386.3662ms
logger=plugins.update.checker t=2023-08-24T10:11:17.4566749Z level=info msg=“Update check succeeded” duration=503.8247ms
logger=cleanup t=2023-08-24T10:21:16.4239598Z level=info msg=“Completed cleanup jobs” duration=8.9545ms
logger=grafana.update.checker t=2023-08-24T10:21:17.1009242Z level=info msg=“Update check succeeded” duration=319.7861ms
logger=plugins.update.checker t=2023-08-24T10:21:17.4403694Z level=info msg=“Update check succeeded” duration=487.8449ms
logger=cleanup t=2023-08-24T10:31:16.4239429Z level=info msg=“Completed cleanup jobs” duration=9.1363ms
logger=grafana.update.checker t=2023-08-24T10:31:17.0920463Z level=info msg=“Update check succeeded” duration=310.8967ms
logger=plugins.update.checker t=2023-08-24T10:31:17.4429569Z level=info msg=“Update check succeeded” duration=490.9707ms
logger=cleanup t=2023-08-24T10:41:16.4244262Z level=info msg=“Completed cleanup jobs” duration=9.356ms
logger=grafana.update.checker t=2023-08-24T10:41:17.1037936Z level=info msg=“Update check succeeded” duration=322.5811ms
logger=plugins.update.checker t=2023-08-24T10:41:17.453813Z level=info msg=“Update check succeeded” duration=501.2014ms
logger=context userId=0 orgId=0 uname= t=2023-08-24T10:49:34.8086533Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=310.9µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context userId=0 orgId=0 uname= t=2023-08-24T10:49:34.9577871Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T10:49:34.9579154Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=1 duration=1.2985ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context userId=0 orgId=0 uname= t=2023-08-24T10:49:35.1116081Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=349.6µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=oauth t=2023-08-24T10:49:35.4199632Z level=info msg=“state check” queryState=35af0a4b9c4d7a2c26d99ae5df1a4707a7c182d856412fb04d27bdc99d3661b6 cookieState=35af0a4b9c4d7a2c26d99ae5df1a4707a7c182d856412fb04d27bdc99d3661b6
logger=http.server t=2023-08-24T10:49:35.5278546Z level=info msg=“Successful Login” User=user@xxxxxxxx.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T10:49:35.5279187Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=108 duration=108.2581ms size=32 referer= handler=/login/:name
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T10:49:40.7342041Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=568.1µs size=0 referer= handler=/api/live/ws
logger=cleanup t=2023-08-24T10:51:16.4244127Z level=info msg=“Completed cleanup jobs” duration=8.942ms
logger=grafana.update.checker t=2023-08-24T10:51:17.0985662Z level=info msg=“Update check succeeded” duration=317.7858ms
logger=plugins.update.checker t=2023-08-24T10:51:17.4428713Z level=info msg=“Update check succeeded” duration=490.4095ms
logger=cleanup t=2023-08-24T11:01:16.4337493Z level=info msg=“Completed cleanup jobs” duration=19.1128ms
logger=grafana.update.checker t=2023-08-24T11:01:17.072396Z level=info msg=“Update check succeeded” duration=291.5541ms
logger=plugins.update.checker t=2023-08-24T11:01:17.5340116Z level=info msg=“Update check succeeded” duration=581.5912ms
logger=context userId=2 orgId=1 uname=user@xxxxxxxx.com t=2023-08-24T11:08:58.5979426Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=84 duration=84.9283ms size=183 referer=https://k8sdbtest1.xxxxxxxx.com/grafana/profile handler=/logout
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:09:14.7814347Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=341.2µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:09:14.9355487Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:09:14.9356063Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=1 duration=1.0967ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:09:15.0869748Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=346.6µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=oauth t=2023-08-24T11:09:15.3892574Z level=info msg=“state check” queryState=8d9000ab8b78a541b8c429aba52356fc65610b00d475946545923b6ac846ba61 cookieState=8d9000ab8b78a541b8c429aba52356fc65610b00d475946545923b6ac846ba61
logger=http.server t=2023-08-24T11:09:15.4551556Z level=info msg=“Successful Login” User=dips@example.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:09:15.4552135Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=66 duration=66.3393ms size=32 referer= handler=/login/:name
logger=context userId=3 orgId=1 uname=dips@example.com t=2023-08-24T11:09:19.3432777Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=683.4µs size=0 referer= handler=/api/live/ws
logger=context userId=3 orgId=1 uname=dips@example.com t=2023-08-24T11:09:39.1195566Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=58 duration=58.1761ms size=183 referer=https://k8sdbtest1.xxxxxxxx.com/grafana/profile handler=/logout
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:10:32.5320013Z level=info msg=“Request Completed” method=GET path= status=302 remote_addr=10.40.101.88 time_ms=0 duration=353.8µs size=37 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:10:32.6861944Z level=info msg=“OAuth auto login enabled. Redirecting to /grafana/login/generic_oauth”
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:10:32.6862405Z level=info msg=“Request Completed” method=GET path=/login status=307 remote_addr=10.40.101.88 time_ms=1 duration=1.1833ms size=64 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:10:32.8463777Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=0 duration=348.8µs size=308 referer=https://k8sdbtest1.xxxxxxxx.com/xxxx/platform/dashboard handler=/login/:name
logger=oauth t=2023-08-24T11:10:33.150351Z level=info msg=“state check” queryState=8ab7af5ac5ca177b7f3f6dca235fe22f6315cddbb5d4af84621067768c9a8a01 cookieState=8ab7af5ac5ca177b7f3f6dca235fe22f6315cddbb5d4af84621067768c9a8a01
logger=http.server t=2023-08-24T11:10:33.2514845Z level=info msg=“Successful Login” User=dips@example.com
logger=context userId=0 orgId=0 uname= t=2023-08-24T11:10:33.2515609Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.40.101.88 time_ms=101 duration=101.5401ms size=32 referer= handler=/login/:name
logger=context userId=3 orgId=1 uname=dips@example.com t=2023-08-24T11:10:36.9350025Z level=info msg=“Request Completed” method=GET path=/api/live/ws status=-1 remote_addr=10.40.101.88 time_ms=0 duration=669.5µs size=0 referer= handler=/api/live/ws
logger=context userId=3 orgId=1 uname=dips@example.com t=2023-08-24T11:10:47.6796708Z level=info msg=“Request Completed” method=GET path=/logout status=302 remote_addr=10.40.101.88 time_ms=44 duration=44.1033ms size=183 referer=“https://k8sdbtest1.xxxxxxxx.com/grafana/?orgId=1” handler=/logout
logger=cleanup t=2023-08-24T11:11:16.4340875Z level=info msg=“Completed cleanup jobs” duration=19.173ms
logger=grafana.update.checker t=2023-08-24T11:11:17.2310503Z level=info msg=“Update check succeeded” duration=450.2846ms
logger=plugins.update.checker t=2023-08-24T11:11:17.4483723Z level=info msg=“Update check succeeded” duration=495.6803ms
logger=cleanup t=2023-08-24T11:21:16.4250375Z level=info msg=“Completed cleanup jobs” duration=9.1938ms
logger=grafana.update.checker t=2023-08-24T11:21:17.1223784Z level=info msg=“Update check succeeded” duration=340.9322ms
logger=plugins.update.checker t=2023-08-24T11:21:17.447676Z level=info msg=“Update check succeeded” duration=495.3705ms
logger=cleanup t=2023-08-24T11:31:16.4250186Z level=info msg=“Completed cleanup jobs” duration=9.4775ms
logger=grafana.update.checker t=2023-08-24T11:31:17.1118565Z level=info msg=“Update check succeeded” duration=330.9005ms
logger=plugins.update.checker t=2023-08-24T11:31:17.4600708Z level=info msg=“Update check succeeded” duration=507.4452ms
logger=cleanup t=2023-08-24T11:41:16.4239127Z level=info msg=“Completed cleanup jobs” duration=9.1748ms
logger=grafana.update.checker t=2023-08-24T11:41:17.1231163Z level=info msg=“Update check succeeded” duration=342.1275ms
logger=plugins.update.checker t=2023-08-24T11:41:17.4754533Z level=info msg=“Update check succeeded” duration=523.158ms
logger=cleanup t=2023-08-24T11:51:16.4349104Z level=info msg=“Completed cleanup jobs” duration=20.0934ms
logger=grafana.update.checker t=2023-08-24T11:51:17.1453275Z level=info msg=“Update check succeeded” duration=364.6835ms
logger=plugins.update.checker t=2023-08-24T11:51:17.4434116Z level=info msg=“Update check succeeded” duration=491.471ms
logger=cleanup t=2023-08-24T12:01:16.4195044Z level=info msg=“Completed cleanup jobs” duration=4.0854ms
logger=grafana.update.checker t=2023-08-24T12:01:17.124413Z level=info msg=“Update check succeeded” duration=343.5935ms
logger=plugins.update.checker t=2023-08-24T12:01:17.443067Z level=info msg=“Update check succeeded” duration=490.9714ms
logger=cleanup t=2023-08-24T12:11:16.4340402Z level=info msg=“Completed cleanup jobs” duration=19.1805ms
logger=grafana.update.checker t=2023-08-24T12:11:17.1330942Z level=info msg=“Update check succeeded” duration=352.5796ms
logger=plugins.update.checker t=2023-08-24T12:11:17.4543731Z level=info msg=“Update check succeeded” duration=501.8531ms