JSON API datasource 504 error

Grafana Configuration
1

  • What Grafana version and what operating system are you using?
    9.5.2

  • What are you trying to achieve?
    Trying to connect json api datasource

  • How are you trying to achieve it?
    Fill in the URL of json api URL and skip tls verify

  • What happened?
    Replied with 504 error from log files, but curled looks good:

[obs_logmo_prod@vshgms103 (prod) log]$ time curl -k -I https://10.17.28.7:10000/api/v1/cmdb/hosts
HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.9.16
Date: Fri, 10 Nov 2023 01:54:30 GMT
Content-Type: application/json
Content-Length: 271218
Connection: close


real    0m0.026s
user    0m0.011s
sys     0m0.002s

  • What did you expect to happen?
    Connection could be set up successfully

  • Can you copy/paste the configuration(s) that you are having problems with?
    In defaults.ini, I set the dialTimeout and tls_handshake_timeout_seconds as 30s.

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.

    504
    504873×969 40.1 KB

And from grafan.log:

logger=data-proxy-log userId=2 orgId=1 uname=dujas path=/api/datasources/proxy/uid/f3f4e572-619c-48f0-85b8-9697a220633f remote_addr=10.17.41.198 referer=https://obs.ai.ms.com.cn/datasources/edit/f3f4e572-619c-48f0-85b8-9697a220633f t=2023-11-10T09:47:02.966160926+08:00 level=error msg="Proxy request timed out" err="net/http: TLS handshake timeout"
logger=context userId=2 orgId=1 uname=dujas t=2023-11-10T09:47:02.966289004+08:00 level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/f3f4e572-619c-48f0-85b8-9697a220633f status=504 remote_addr=10.17.41.198 time_ms=30006 duration=30.006467657s size=0 referer=https://obs.ai.ms.com.cn/datasources/edit/f3f4e572-619c-48f0-85b8-9697a220633f handler=/api/datasources/proxy/uid/:uid
2

And once I set the tls_handshake_timeout to 300, the error code turned to be 503 with error “JSON API: Service Unavailable”:

logger=context userId=2 orgId=1 uname=dujas t=2023-11-10T10:30:59.354153774+08:00 level=error msg="Request Completed" method=GET path=/api/datasources/proxy/uid/f3f4e572-619c-48f0-85b8-9697a220633f status=503 remote_addr=10.17.41.198 time_ms=75060 duration=1m15.060265005s size=1953 referer=https://obs.ai.ms.com.cn/datasources/edit/f3f4e572-619c-48f0-85b8-9697a220633f handler=/api/datasources/proxy/uid/:uid

3

Issue got fixed after converting the IP to hostname.