Error: Status: 504:Message: Get "http://Loki-querier / Get "http://thanos-querier"

Grafana Configuration
1

Hello, I’m new to this Grafana community, thank you very much for sharing.

I have deployed Grafana 10.1.4 on Kubernetes with Loki and Thanos as datasource. But for several days now, every time I run a query that is longer than 2 days, I get this error Status: 504 timeout awaiting response headers (Client.Timeout exceeded while awaiting headers).

Some forums suggest that I adjust the timeout of my datasources in the Grafana configuration, but I’m still not sure about doing it, because I don’t know if I should adjust it in each datasource or in my Grafana.ini?

I really appreciate any help you can give me.

This is the configuration of my datasources:

    "datasources": [
        {
             "access": "proxy",
             "editable": true,
             "name": "prometheus",
             "orgId": 1,
             "type": "prometheus",
             "uid": "prometheus",
             "url": http://thanos-querier.{{  .Values.global.namespace }}.svc:9090{{  .Values.global.monitoring.thanos.querier.path }},
             "version": 1,
             "jsonData": {"exemplarTraceIdDestinations":[{"nombre":"TraceID","datasourceUid":"tempo"}]}
        },
        {
             "access": "proxy",
             "editable": true,
             "name": "loki",
             "orgId": 1,
             "type": "loki",
             "uid": "loki",
             "url": http://loki-distributed-gateway.{{  .Values.global.namespace }}.svc:8080,
             "version": 1,
             "jsonData":{"maxLines":"5000","derivedFields":[{"datasourceUid":"tempo","matcherRegex":"traceId: (\\w+)","name":"TraceId","url":'$${__value.raw}'}]}
        },
2

Hello,

If your question is about: “Should I change the timeout in Grafana or in the datasources”, I think the answer is: you should change the timeout in grafana.ini.