- What Grafana version and what operating system are you using?
Grafana v12.1.1 (df5de8219b)
- What are you trying to achieve?
Get traces through a Jaeger datasource
- How are you trying to achieve it?
Configuration via Grafana UI
- What happened?
The datasource gets configured, the save&test passes, and I can fetch the list of Service Names and Operation Names from Jaeger. However, once I select a Service and an Operation and click “Run Query”, I get a `400 Bad Request` error.
- What did you expect to happen?
The datasource correctly running
- Can you copy/paste the configuration(s) that you are having problems with?
The datasource is configured as follows:
{
"id": 11,
"uid": "[REDACTED]",
"orgId": 1,
"name": "jaeger-debug",
"type": "jaeger",
"typeLogoUrl": "public/plugins/jaeger/img/jaeger_logo.svg",
"access": "proxy",
"url": "[REDACTED]",
"user": "",
"database": "",
"basicAuth": false,
"basicAuthUser": "",
"withCredentials": false,
"isDefault": false,
"jsonData": {
"nodeGraph": {
"enabled": true
},
"pdcInjected": false,
"tlsSkipVerify": true,
"tracesToMetrics": {
"datasourceUid": "[REDACTED]"
}
},
"secureJsonFields": {},
"version": 5,
"readOnly": false,
"accessControl": {
"alert.instances.external:read": true,
"alert.instances.external:write": true,
"alert.notifications.external:read": true,
"alert.notifications.external:write": true,
"alert.rules.external:read": true,
"alert.rules.external:write": true,
"datasources.id:read": true,
"datasources:delete": true,
"datasources:query": true,
"datasources:read": true,
"datasources:write": true
},
"apiVersion": ""
}
This is a cURL command I copied from the browser’s Developer Console to replicate the failing request:
curl 'http://[GRAFANA_URL]/api/ds/query?ds_type=jaeger&requestId=explore_r5i' \
-H 'Accept-Language: en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7' \
-H 'Connection: keep-alive' \
-b 'grafana_session=[REDACTED]; grafana_session_expiry=[REDACTED]' \
-H 'Origin: http://[GRAFANA_URL]' \
-H 'Referer: http://[GRAFANA_URL]/explore?...' \
-H 'User-Agent: Mozilla/5.0 ...' \
-H 'accept: application/json, text/plain, */*' \
-H 'content-type: application/json' \
-H 'x-cache-skip: true' \
-H 'x-datasource-uid: [REDACTED]' \
-H 'x-grafana-device-id: [REDACTED]' \
-H 'x-grafana-org-id: 1' \
-H 'x-plugin-id: jaeger' \
--data-raw '{"queries":[{"refId":"A","datasource":{"type":"jaeger","uid":"[REDACTED]"},"queryType":"search","service":"[REDACTED]","operation":"","minDuration":"","maxDuration":"","datasourceId":10,"intervalMs":2000,"maxDataPoints":1365}],"from":"[REDACTED]","to":"[REDACTED]"}' \
--insecure
I get this as response:
{"results":{"A":{"error":"request failed: 404 Not Found","errorSource":"downstream","status":500}}}
I can’t honestly figure out if it’s a configuration issue on my end, a bug, or some incompatibility between Grafana and Jaeger versions. I know Jaeger V1 has been deprecated for several months now, but I still need to use it at this time.
Moreover, I can see that the API call never reaches Jaeger, or rather, the reverse proxy in front of it: it seems like Grafana does not issue the API call at all.
For comparison, an older version (v10.4.1) of Grafana successfully retrieves traces from the very same Jaeger instance. Here is the datasource JSON for the working configuration:
{
"id": 83,
"uid": "[REDACTED]",
"orgId": 1,
"name": "jaeger-debug",
"type": "jaeger",
"typeLogoUrl": "public/app/plugins/datasource/jaeger/img/jaeger_logo.svg",
"access": "proxy",
"url": "[REDACTED]",
"user": "",
"database": "",
"basicAuth": false,
"basicAuthUser": "",
"withCredentials": false,
"isDefault": false,
"jsonData": {
"nodeGraph": {
"enabled": true
},
"tlsSkipVerify": true,
"tracesToMetrics": {
"datasourceUid": "[REDACTED]"
}
},
"secureJsonFields": {},
"version": 4,
"readOnly": false,
"accessControl": {
"alert.instances.external:read": true,
"alert.instances.external:write": true,
"alert.notifications.external:read": true,
"alert.notifications.external:write": true,
"alert.rules.external:read": true,
"alert.rules.external:write": true,
"datasources.caching:read": true,
"datasources.caching:write": true,
"datasources.id:read": true,
"datasources.permissions:read": true,
"datasources.permissions:write": true,
"datasources:delete": true,
"datasources:query": true,
"datasources:read": true,
"datasources:write": true
}
}
And this is the cURL for the working request which gets an HTTP 200 with traces as expected:
curl 'https://[REDACTED]/api/datasources/proxy/uid/[REDACTED]/api/traces?service=[REDACTED]&start=[REDACTED]&end=[REDACTED]&lookback=custom' \
-H 'Accept-Language: en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7' \
-H 'Connection: keep-alive' \
-b '[REDACTED]' \
-H 'Referer: https://[REDACTED]/explore?...' \
-H 'Sec-Fetch-Dest: empty' \
-H 'Sec-Fetch-Mode: cors' \
-H 'Sec-Fetch-Site: same-origin' \
-H 'User-Agent: Mozilla/5.0 ...' \
-H 'accept: application/json, text/plain, */*' \
-H 'sec-ch-ua: "Chromium";v="140", "Not=A?Brand";v="24", "Microsoft Edge";v="140"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "[REDACTED]"' \
-H 'x-grafana-device-id: [REDACTED]' \
-H 'x-grafana-org-id: 1'
I can see the older Grafana call is quite different from the newer one, which makes sense given there are two major versions between them.
- Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Grafana does not show errors, even with log level set to DEBUG
- Did you follow any online instructions? If so, what is the URL?
