Token Errors with Generic OAuth (OIDC) in Openshift with Istio

Grafana Authentication
,
1

Deploying Grafana 10.1.0 in Openshift 4.10 using the bitnami container image.

Currently attempting to setup generic oauth following the settings here and receving

Login failed Failed to get token from provider

in the browser and seeing the following in the logs

logger=context userId=0 orgId=0 uname= t=2023-10-18T16:25:11.077613197Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.129.12.1 time_ms=0 duration=97.953µs size=419 referer=https://grafana.example.com/login handler=/login/:name
logger=authn.service t=2023-10-18T16:25:12.312923754Z level=warn msg=“Failed to authenticate request” client=auth.client.generic_oauth error=“[auth.oauth.token.exchange] failed to exchange code to token: Post "https://oauthexample.com/as/token.oauth2\”: http: server gave HTTP response to HTTPS client"
logger=context userId=0 orgId=0 uname= t=2023-10-18T16:25:12.386450633Z level=info msg=“Request Completed” method=GET path=/login/generic_oauth status=302 remote_addr=10.129.12.1 time_ms=75 duration=75.506247ms size=29 referer=https://grafana.example.com/ handler=/login/:name

Grafana settings:
GF_SERVER_PROTOCOL=http
GF_SERVER_ROOT_URL=https://grafana.example.com

Oauth Settings:
Oauth redirect uri = https://grafana.example.com/login/generic_oauth

Istio Config:

  - apiVersion: networking.istio.io/v1alpha3
    kind: VirtualService
    metadata:
      labels:
        app: ${APP}-grafana
      name: https
      namespace: ${OPENSHIFT_PROJECT}
    spec:
      hosts:
        - ${OPENSHIFT_HTTPS_HOSTNAME}
      gateways:
        - ${APP}-grafana
      http:
        - route:
            - destination:
                host: ${APP}-grafana
                port:
                  number: 443

  - apiVersion: networking.istio.io/v1alpha3
    kind: Gateway
    metadata:
      labels:
        app: ${APP}-grafana
      name: ${APP}-grafana
      namespace: ${OPENSHIFT_PROJECT}
    spec:
      servers:
      - hosts:
          - ${OPENSHIFT_HTTPS_HOSTNAME}
        port:
          name: https
          number: 443
          protocol: HTTPS
        tls:
          credentialName: ${OPENSHIFT_HTTPS_HOSTNAME_CERT_SECRET}
          mode: SIMPLE
      - hosts:
          - ${OPENSHIFT_HTTPS_HOSTNAME}
        port:
          name: http
          number: 80
          protocol: HTTP
        tls:
          httpsRedirect: true
      selector:
        istio: ingressgateway

kind: Service
apiVersion: v1
metadata:
  name: ${APP}-grafana
  namespace: ${OPENSHIFT_PROJECT}
  labels:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    helm.sh/chart: grafana-9.1.0
  annotations:
    meta.helm.sh/release-name: grafana
    meta.helm.sh/release-namespace: grafana
spec:
  clusterIP: <ip_address>
  ipFamilies:
    - IPv4
  ports:
    - name: http
      protocol: TCP
      port: 443
      targetPort: 3000
  internalTrafficPolicy: Cluster
  clusterIPs:
    - <ip_address>
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app.kubernetes.io/component: grafana
    app.kubernetes.io/instance: grafana
    app.kubernetes.io/name: grafana
status:
  loadBalancer: {}

I confirmed with the team that manages our auth services that there are no errors on their end and it seems to be sending the correct information back to the pod and I can get to grafana and login with a local account.

I believe the issues is somewhere between the root_url and the istio configuration (tls termination). I’ve tried several combinations including setting up the container with https, but get the same http: server gave HTTP response to HTTPS client error, but in the browser.

2

That looks like https://oauthexample.com doesn’t work with HTTPS (TLS) properly.

3

That looks like https://oauthexample.com doesn’t work with HTTPS (TLS) properly.

I trust that everything is working fine on their end. They are an enterprise wide team and we have it working on our deployment to EC2 instances. The only difference is the EC2 deployment doesn’t terminate tls after the load balancer and by extension isn’t in OpenShift.

4

I wouldn’t trust. I would verify first.

5

Hi, have you managed to solve this problem? I also encountered this behavior of grafana when integrating with authentik (this error occurs even with the new Grafana installation…)