Unauthorized/Forbidden while accessing OnCall engine: /api/internal/v1/plugin/v2/status, status code: 403, check token
2024-08-30 03:02:04 source=engine:app google_trace_id=none logger=django.request Forbidden: /api/internal/v1/plugin/v2/status
Oncall Plugin version : 1.9.17
opened 08:39AM - 30 Aug 24 UTC
closed 01:24PM - 30 Nov 24 UTC
bug
part:alert flow & configuration
### What went wrong?
Hello,
I am currently using Grafana OSS version 11.1.0 an… d have deployed the Grafana OnCall plugin(1.9.15).
**What happened**:
- When attempting to connect the OnCall plugin, I encounter the following error message: 'Plugin is not connected Unauthorized/Forbidden while accessing OnCall engine: /api/internal/v1/plugin/v2/status, status code: 403, check token.'
grafana.log
```
logger=context userId=2 orgId=1 uname=sa-1-sa-autogen-oncall t=2024-08-30T08:34:02.990990635Z level=info msg="Request Completed" method=GET path=/api/plugins/grafana-incident-app/settings status=404 remote_addr=10.201.36.203 time_ms=39 duration=39.011119ms size=64 referer= handler=/api/plugins/:pluginId/settings status_source=server
logger=plugin.grafana-oncall-app t=2024-08-30T08:34:02.992883818Z level=error msg="getting incident plugin settings" error="request did not return 200: 404"
logger=context userId=2 orgId=1 uname=sa-1-sa-autogen-oncall t=2024-08-30T08:34:03.054445886Z level=info msg="Request Completed" method=GET path=/api/plugins/grafana-labels-app/settings status=404 remote_addr=10.201.36.203 time_ms=38 duration=38.319328ms size=64 referer= handler=/api/plugins/:pluginId/settings status_source=server
logger=plugin.grafana-oncall-app t=2024-08-30T08:34:03.056015885Z level=error msg="getting labels plugin settings" error="request did not return 200: 404"
logger=plugin.grafana-oncall-app t=2024-08-30T08:34:03.126314798Z level=info msg=GetSyncData time=65
logger=plugin.grafana-oncall-app t=2024-08-30T08:34:03.175113158Z level=error msg="Error unmarshalling OnCallError" error="invalid character '<' looking for beginning of value"
logger=plugin.grafana-oncall-app t=2024-08-30T08:34:03.205154621Z level=info msg=GetUser user="map[Email:admin@localhost Login:admin Name:admin Role:Admin]"
```
engine log
```
2024-08-30 08:32:41 source=engine:app google_trace_id=none logger=root inbound latency=0.000621 status=200 method=GET path=/api/internal/v1/health/ user_agent=Go-http-client/1.1 content-length=0 slow=0
2024-08-30 08:32:41 source=engine:uwsgi status=200 method=GET path=/api/internal/v1/health/ latency=0.002093 google_trace_id=- protocol=HTTP/1.1 resp_size=221 req_body_size=0
2024-08-30 08:32:41 source=engine:app google_trace_id=none logger=apps.social_auth.middlewares SocialAuthAuthCanceledExceptionMiddleware.process_exception: Object of type OnCallError is not JSON serializable
2024-08-30 08:32:41 source=engine:app google_trace_id=none logger=django.request Internal Server Error: /api/internal/v1/plugin/v2/install
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/django/core/handlers/base.py", line 220, in _get_response
response = response.render()
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/django/template/response.py", line 114, in render
self.content = self.rendered_content
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rest_framework/response.py", line 74, in rendered_content
ret = renderer.render(self.data, accepted_media_type, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rest_framework/renderers.py", line 100, in render
ret = json.dumps(
^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rest_framework/utils/json.py", line 25, in dumps
return json.dumps(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/__init__.py", line 238, in dumps
**kw).encode(obj)
^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/encoder.py", line 200, in encode
chunks = self.iterencode(o, _one_shot=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/encoder.py", line 258, in iterencode
return _iterencode(o, 0)
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/rest_framework/utils/encoders.py", line 67, in default
return super().default(obj)
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/json/encoder.py", line 180, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type OnCallError is not JSON serializable
```
The configuration(oncall values.yaml) is as follows, and all pods are running normally.
```
base_url: grafana-oncall-test.xxx.xx
base_url_protocol: https
# Celery workers pods configuration
celery:
replicaCount: 1
worker_queue: "default,critical,long,slack,telegram,webhook,celery,grafana,retry"
worker_concurrency: "1"
worker_max_tasks_per_child: "100"
worker_beat_enabled: "True"
worker_shutdown_interval: "65m"
livenessProbe:
enabled: true
initialDelaySeconds: 30
periodSeconds: 300
timeoutSeconds: 10
oncall:
devMode: false
secrets:
existingSecret: oncall-secret
secretKey: secretKey
mirageSecretKey: mirageSecretKey
slack:
enabled: false
telegram:
enabled: false
smtp:
enabled: false
exporter:
enabled: false
migrate:
enabled: true
useHook: false
ingress:
enabled: true
className: traefik-default
tls:
- hosts:
- "{{ .Values.base_url }}"
secretName: xxx-xx-tls
ingress-nginx:
enabled: false
cert-manager:
enabled: false
database:
type: postgresql
mariadb:
enabled: true
postgresql:
#enabled: false
enabled: true
auth:
database: oncall
existingSecret: postgresql-secret
rabbitmq:
enabled: true
auth:
existingPasswordSecret: rabbitmq-secret
broker:
type: rabbitmq
redis:
enabled: true
auth:
existingSecret:
grafana:
enabled: true
grafana.ini:
server:
domain: grafana-oncall-test.xxx.xx
root_url: https://grafana-oncall-test.xxxx.xx/grafana #"%(protocol)s://%(domain)s/grafana"
serve_from_sub_path: true
cert_file: /etc/grafana/tls.crt
cert_key: /etc/grafana/tls.key
persistence:
enabled: true
rbac:
pspEnabled: false
plugins:
- grafana-oncall-app
extraSecretMounts:
- name: tls-cert
mountPath: /etc/grafana/tls.crt
secretName: xxx-xx-tls
readOnly: true
subPath: tls.crt
- name: tls-key
mountPath: /etc/grafana/tls.key
secretName: xxx-xx-tls
readOnly: true
subPath: tls.key
image:
repository: grafana/grafana
initChownData:
image:
repository: library/busybox
testFramework:
image:
repository: bats/bats
datasources:
oncall.yaml:
apiVersion: 1
apps:
- type: grafana-oncall-app
org_id: 1
enabled: true
jsonData:
stackId: 5
orgId: 1
onCallApiUrl: http://grafana-oncall-test-engine:8080
prometheus:
enabled: false
```
Here are the steps I followed to add the plugin in Grafana:
1. Enabled the plugin
2. Registered the OnCall API URL
3. Tested the connection using the 'Connect' button
4. However, when testing the connection with the 'Connect' button, a 403 error occurs.
I have some additional questions: **How does the Service Account created when enabling the plugin in Grafana interact with OnCall**?
I believe the Service Account generated by Grafana needs to be mounted as a secret in OnCall. Which environment variable should be used to assign Grafana’s Service Account in OnCall?
And most importantl**y, I cannot find a way to view the actual token value of the Service Account automatically created by Grafana**.
Do you know how to resolve this issue?
### How do we reproduce it?
1. Enabled the plugin in grafana
2. Registered the OnCall API URL
3. Tested the connection using the 'Connect' button
A 403 error occurs.
### Grafana OnCall Version
1.9.15
### Product Area
Alert Flow & Configuration
### Grafana OnCall Platform?
Kubernetes
### User's Browser?
chrom 128.0.6613.114
### Anything else to add?
_No response_
scroll down and see how can fix.
Not exactly sure when this started but I am seeing Loading OnCall integration failed message when I edit either one of my configured contact points. I do not have the OnCall Plugin installed nor have I tried to configure it.
Current Grafana version:12.4.1 running on Debian Trixie 13. (raspberry pi 5)
Still seeing error message after Grafana upgrade to (Grafana v12.4.2 (ebade4c739)).
No longer seeing message (Loading OnCall intergration failed) when editiing contact points after upgrading to Grafana v13.0.0 (e9de482b).
