Github Auth - Server error on Login

Grafana Authentication
1

What Grafana version are you using?

v5.1.1

What datasource are you using?

Prometheus

What OS are you running grafana on?

CentOS

What did you do?

Configured github authentication using environment variables

What was the expected result?

User should login via github button

What happened instead?

After confirming oauth access in github’s flow, user is redirected to grafana endpoint /login/github which then throws a server error

31%20PM
31%20PM987×352 29.5 KB

Include raw network request & response: get by opening Chrome Dev Tools (F12, Ctrl+Shift+I on windows, Cmd+Opt+I on Mac), go the network tab.

44%20PM
44%20PM1221×575 91.4 KB

Server logs show the following:

t=2018-05-08T19:47:57+0000 lvl=info msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=302 remote_addr=65.214.129.74 time_ms=0 size=299 referer=example.com
t=2018-05-08T19:47:57+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="oauth2: server response missing access_token"
t=2018-05-08T19:47:57+0000 lvl=eror msg="Request Completed" logger=context userId=0 orgId=0 uname= method=GET path=/login/github status=500 remote_addr=65.214.129.74 time_ms=224 size=1724 referer=example.com
t=2018-05-08T19:48:02+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0
t=2018-05-08T19:48:12+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=0

Environment Configuration based on kubernetes grafana.yaml file

env:
        - name: GF_SERVER_ROOT_URL
          value: some root url
        - name: GF_LOG_MODE
          value: "console file syslog"
        - name: GF_LOG_LEVEL
          value: "debug"
        - name: GF_AUTH_GITHUB_ENABLED
          value: "true"
        - name: GF_AUTH_GITHUB_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: github-credentials
              key: id
        - name: GF_AUTH_GITHUB_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: github-credentials
              key: secret
        - name: GF_AUTH_GITHUB_ALLOW_SIGN_UP
          value: "true"
        - name: GF_AUTH_GITHUB_SCOPES
          value: "user:email,read:org"
        - name: GF_AUTH_GITHUB_AUTH_URL
          value: default
        - name: GF_AUTH_GITHUB_TOKEN_URL
          value: default
        - name: GF_AUTH_GITHUB_API_URL
          value: default
        - name: GF_AUTH_GITHUB_ALLOWED_ORGANIZATIONS
          value: some org
        - name: GF_AUTH_GITHUB_TLS_SKIP_VERIFY_INSECURE
          value: "true"
        - name: GF_SECURITY_ADMIN_USER
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: user
        - name: GF_SECURITY_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: grafana-credentials
              key: password
        - name: GF_SMTP_ENABLED
          value: "true"
        - name: GF_SMTP_HOST
          value: some smtp host
        - name: GF_SMTP_USER
          valueFrom:
            secretKeyRef:
              name: smtp-credentials
              key: username
        - name: GF_SMTP_PASSWORD
          valueFrom:
            secretKeyRef:
              name: smtp-credentials
              key: password
        - name: GF_SMTP_FROM_ADDRESS
          value: some email
        - name: GF_SMTP_FROM_NAME
          value: some name

I am running this as part of a kubernetes deployment configuration based on this repo camilb/prometheus-kubernetes

I am also running an elb nginx ingress in front with ssl termination at the elb to enable https traffic routed through Route 53 in AWS.

Let me know if there is anymore information i can provide

2

This error indicates that the response from the Github API did not include a valid access token, but I’m not sure why that would happen.

3

hello @dcech, I updated my original post with a bit more of my configuration data.

This problem started occuring after I upgraded from 4.6.3 to 5.1.1. I originally upgraded because of the github oauth bug regarding organizations present in earlier versions before 5.0.

Any guidance would be awesome

Thanks,

Chance

4

Did you configure Authorization callback URL for your test/dev Grafana domain+path?
You may try to exchange obtained code for a token with curl to see response from GitHub directly - error message should be there. Curl example:

curl --request POST \
  --url 'https://github.com/login/oauth/access_token' \
  --header 'content-type: application/json' \
  --data '{"grant_type":"authorization_code","client_id": "YOUR_CLIENT_ID","client_secret": "YOUR_CLIENT_SECRET","code": "YOUR_AUTHORIZATION_CODE","redirect_uri": "https://YOUR_APP/callback"}'

Doc: https://developer.github.com/apps/building-oauth-apps/authorization-options-for-oauth-apps/#2-users-are-redirected-back-to-your-site-by-github

5

Thanks for the help @jangaraj

The callback url in github is configured with the following:
https://dev-grafana.example.com/login/github

Here is the response to that curl command
access_token=8871f96d48c710533883221f94062cd2f3e1056a&scope=read%3Aorg%2Cuser%3Aemail&token_type=bearer

6

GitHub response looks good to me.

It looks like it’s failing here: https://github.com/golang/oauth2/blob/6881fee410a5daf86371371f9ad451b95e168b71/internal/token.go#L256 (test has been added 4 months ago). It’s not able to parse access_token. I don’t know why. You may try to add some debugging lines into Go code to see raw GitHub response and parsed access_token values, just to debug golang/oauth2 RetrieveToken() function. Of course, you will need to build your own Grafana binary.

7

After looking into that code in the oauth2 library I did notice that the response header for content-type does not full text match the expected cases inside the oauth2 package.

Response headers

HTTP/1.1 200 OK
Date: Wed, 09 May 2018 23:10:38 GMT
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Transfer-Encoding: chunked
Server: GitHub.com
Status: 200 OK
Cache-Control: no-cache
Vary: X-PJAX
X-Request-Id: eefd32c7-0dbd-4e27-b95f-9df7409773c8
X-Runtime: 0.015145
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Expect-CT: max-age=2592000, report-uri="https://api.github.com/_private/browser/errors"
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; connect-src 'self' uploads.github.com status.github.com collector.githubapp.com api.github.com www.google-analytics.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com wss://live.github.com; font-src assets-cdn.github.com; form-action 'self' github.com gist.github.com; frame-ancestors 'none'; frame-src render.githubusercontent.com; img-src 'self' data: assets-cdn.github.com identicons.github.com collector.githubapp.com github-cloud.s3.amazonaws.com *.githubusercontent.com; manifest-src 'self'; media-src 'none'; script-src assets-cdn.github.com; style-src 'unsafe-inline' assets-cdn.github.com
X-Runtime-rack: 0.021266
Vary: Accept-Encoding
X-GitHub-Request-Id: C1E6:0AB1:D3A4D:14AAC2:5AF37FEE

Oauth2 expected content-type values:

content, _, _ := mime.ParseMediaType(r.Header.Get("Content-Type"))
switch content {
case "application/x-www-form-urlencoded", "text/plain":

Notice that the response header contains the charset for content-type? I’m not sure if go’s mime library correctly parses out that data string before it does the comparison in the switch case.

I am also totally bewildered by the fact that no one else seems to be experiencing this issue when using github login.

8

That is not a problem - https://play.golang.org/p/NWNEqG2DdQX

9

I am running into the same problem as described by @chancelarkst.

My grafana is sitting behind an instance of HAProxy. I’ve verified with curl that I can obtain a token from github. This problem seems present in older versions as well as new ones. The issue is occurring when the docker container running grafana is exposed to the internet and when it is behind a load balancer.

10

I have the same problem with grafana 6.2.2 as described by @chancelarkst

Someone solved this problem?

Sorry, My issue is from my bad. It’s not relative from this issue :slight_smile:

11

I got the same error. I am using ingress-nginx with grafana behind in k8s. I can get access_token using curl, but getting
login.OAuthLogin(missing saved state) inside grafana.
I set cookie_samesite = lax, domain = my.ingress.route, root_url = https://my.ingress.route and cookie_secure = true.
Any updates?

12

@mjmayer Did you solved problem? How?

13

I stuck Grafana behind basic auth and called it a day.