Github Auth - Server error on Login

chancelarkst · May 8, 2018, 8:33pm

What Grafana version are you using?

v5.1.1

What datasource are you using?

Prometheus

What OS are you running grafana on?

CentOS

What did you do?

Configured github authentication using environment variables

What was the expected result?

User should login via github button

What happened instead?

After confirming oauth access in github’s flow, user is redirected to grafana endpoint /login/github which then throws a server error

Include raw network request & response: get by opening Chrome Dev Tools (F12, Ctrl+Shift+I on windows, Cmd+Opt+I on Mac), go the network tab.

Server logs show the following:

t=2018-05-08T19:47:57+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=302 remote_addr=65.214.129.74 time_ms=0 size=299 referer=example.com
t=2018-05-08T19:47:57+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: server response missing access_token"
t=2018-05-08T19:47:57+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=500 remote_addr=65.214.129.74 time_ms=224 size=1724 referer=example.com
t=2018-05-08T19:48:02+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2018-05-08T19:48:12+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0

Environment Configuration based on kubernetes grafana.yaml file

env:
        - name: GF_SERVER_ROOT_URL
          value: some root url
        - name: GF_LOG_MODE
          value: "console file syslog"
        - name: GF_LOG_LEVEL
          value: "debug"
        - name: GF_AUTH_GITHUB_ENABLED
          value: "true"
        - name: GF_AUTH_GITHUB_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: github-credentials
              key: id
        - name: GF_AUTH_GITHUB_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: github-credentials
              key: secret
        - name: GF_AUTH_GITHUB_ALLOW_SIGN_UP
          value: "true"
        - name: GF_AUTH_GITHUB_SCOPES
          value: "user:email,read:org"
        - name: GF_AUTH_GITHUB_AUTH_URL
          value: default
        - name: GF_AUTH_GITHUB_TOKEN_URL
          value: default
        - name: GF_AUTH_GITHUB_API_URL
          value: default
        - name: GF_AUTH_GITHUB_ALLOWED_ORGANIZATIONS
          value: some org
        - name: GF_AUTH_GITHUB_TLS_SKIP_VERIFY_INSECURE
          value: "true"
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        - name: GF_SMTP_ENABLED
          value: "true"
        - name: GF_SMTP_HOST
          value: some smtp host
        - name: GF_SMTP_USER
          valueFrom:
            secretKeyRef:
              name: smtp-credentials
              key: username
        - name: GF_SMTP_PASSWORD
          valueFrom:
            secretKeyRef:
              name: smtp-credentials
              key: password
        - name: GF_SMTP_FROM_ADDRESS
          value: some email
        - name: GF_SMTP_FROM_NAME
          value: some name

I am running this as part of a kubernetes deployment configuration based on this repo camilb/prometheus-kubernetes

I am also running an elb nginx ingress in front with ssl termination at the elb to enable https traffic routed through Route 53 in AWS.

Let me know if there is anymore information i can provide

dcech · May 8, 2018, 8:47pm

This error indicates that the response from the Github API did not include a valid access token, but I’m not sure why that would happen.

chancelarkst · May 8, 2018, 8:51pm

hello @dcech, I updated my original post with a bit more of my configuration data.

This problem started occuring after I upgraded from 4.6.3 to 5.1.1. I originally upgraded because of the github oauth bug regarding organizations present in earlier versions before 5.0.

Any guidance would be awesome

Thanks,

Chance

jangaraj · May 9, 2018, 1:32pm

Did you configure Authorization callback URL for your test/dev Grafana domain+path?
You may try to exchange obtained code for a token with curl to see response from GitHub directly - error message should be there. Curl example:

curl --request POST \
  --url 'https://github.com/login/oauth/access_token' \
  --header 'content-type: application/json' \
  --data '{"grant_type":"authorization_code","client_id": "YOUR_CLIENT_ID","client_secret": "YOUR_CLIENT_SECRET","code": "YOUR_AUTHORIZATION_CODE","redirect_uri": "https://YOUR_APP/callback"}'

Doc: https://developer.github.com/apps/building-oauth-apps/authorization-options-for-oauth-apps/#2-users-are-redirected-back-to-your-site-by-github

chancelarkst · May 9, 2018, 4:48pm

Thanks for the help @jangaraj

The callback url in github is configured with the following:
https://dev-grafana.example.com/login/github

Here is the response to that curl command
access_token=8871f96d48c710533883221f94062cd2f3e1056a&scope=read%3Aorg%2Cuser%3Aemail&token_type=bearer

jangaraj · May 9, 2018, 6:13pm

GitHub response looks good to me.

It looks like it’s failing here: https://github.com/golang/oauth2/blob/6881fee410a5daf86371371f9ad451b95e168b71/internal/token.go#L256 (test has been added 4 months ago). It’s not able to parse access_token. I don’t know why. You may try to add some debugging lines into Go code to see raw GitHub response and parsed access_token values, just to debug golang/oauth2 RetrieveToken() function. Of course, you will need to build your own Grafana binary.

chancelarkst · May 10, 2018, 5:34pm

After looking into that code in the oauth2 library I did notice that the response header for content-type does not full text match the expected cases inside the oauth2 package.

Response headers

HTTP/1.1 200 OK
Date: Wed, 09 May 2018 23:10:38 GMT
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Transfer-Encoding: chunked
Server: GitHub.com
Status: 200 OK
Cache-Control: no-cache
Vary: X-PJAX
X-Request-Id: eefd32c7-0dbd-4e27-b95f-9df7409773c8
X-Runtime: 0.015145
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com
X-Runtime-rack: 0.021266
Vary: Accept-Encoding
X-GitHub-Request-Id: C1E6:0AB1:D3A4D:14AAC2:5AF37FEE

Oauth2 expected content-type values:

content, _, _ := mime.ParseMediaType(r.Header.Get("Content-Type"))
switch content {
case "application/x-www-form-urlencoded", "text/plain":

Notice that the response header contains the charset for content-type? I’m not sure if go’s mime library correctly parses out that data string before it does the comparison in the switch case.

I am also totally bewildered by the fact that no one else seems to be experiencing this issue when using github login.

jangaraj · May 10, 2018, 10:08pm

That is not a problem - https://play.golang.org/p/NWNEqG2DdQX

mjmayer · December 7, 2018, 4:05pm

I am running into the same problem as described by @chancelarkst.

My grafana is sitting behind an instance of HAProxy. I’ve verified with curl that I can obtain a token from github. This problem seems present in older versions as well as new ones. The issue is occurring when the docker container running grafana is exposed to the internet and when it is behind a load balancer.

anarcher · June 18, 2019, 5:06am

~~I have the same problem with grafana 6.2.2 as described by @chancelarkst~~

~~Someone solved this problem?~~

Sorry, My issue is from my bad. It’s not relative from this issue

sergeyshaykhullin · January 24, 2020, 1:46pm

I got the same error. I am using ingress-nginx with grafana behind in k8s. I can get access_token using curl, but getting
login.OAuthLogin(missing saved state) inside grafana.
I set cookie_samesite = lax, domain = my.ingress.route, root_url = https://my.ingress.route and cookie_secure = true.
Any updates?

sergeyshaykhullin · January 24, 2020, 1:53pm

@mjmayer Did you solved problem? How?

mjmayer · January 24, 2020, 3:11pm

I stuck Grafana behind basic auth and called it a day.