Test Loki Datasource in Grafana configuration fails (proxy-related)

On one RHEL 8.4 VM, I install following software (no containerization, simplest startup with shell script

  • Grafana 9.1.2
  • Loki 2.6.1
export HTTP_PROXY=
export HTTPS_PROXY=
/apps/sw/loki/loki -config.file=/apps/sw/loki/loki.yml

[gtos@anr-p-rhl-tosmon01 startup]$ cat /apps/sw/loki/loki.yml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  grpc_listen_address: 127.0.0.1

common:
  path_prefix: /apps/loki
  storage:
    filesystem:
      chunks_directory: /apps/loki/chunks
      rules_directory: /apps/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

ruler:
  alertmanager_url: http://localhost:9093

Create and Test a Loki Datasource in the Grafana Datasources panel fails with the following appearing in the loki logs.

level=warn ts=2022-11-28T12:18:02.605236552Z caller=pool.go:184 msg="removing frontend failing healthcheck" addr=10.203.168.180:9096 reason="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to do connect handshake, response: \\\"HTTP/1.1 307 Proxy Redirect\\\\r\\\\nConnection: close\\\\r\\\\nContent-Length: 1868\\\\r\\\\nCache-Control: no-cache\\\\r\\\\nContent-Type: text/html\\\\r\\\\nDate: Mon, 28 Nov 2022 13:18:01 CET\\\\r\\\\nLocation: https://*****PROXY****-proxy/B0001D0000N0001N0001F0000S0000R0004/10.203.168.180/https://10.203.168.180:9096/\\\\r\\\\nMime-Version: 1.0\\\\r\\\\nProxy-Connection: close\\\\r\\\\n\\\\r\\\\n<!DOCTYPE HTML PUBLIC \\\\\\\"-//W3C//DTD HTML 4.01 Transitional//EN\\\\\\\"\\\\n\\\\\\\"http://www.w3.org/TR/html4/loose.dtd\\\\\\\">\\\\n<html>\\\\n<head>\\\\n<meta http-equiv=\\\\\\\"Content-Type\\\\\\\" content=\\\\\\\"text/html; charset=UTF-8\\\\\\\">\\\\n<title>Notification: Redirect</title>\\\\n<style type=\\\\\\\"text/css\\\\\\\">\\\\nbody {\\\\n font-family: Arial, Helvetica, sans-serif;\\\\n font-size: 14px;\\\\n color:#333333;\\\\n background-color: #ffffff;\\\\n}\\\\nh1 {\\\\n font-size: 18px;\\\\n font-weight: bold;\\\\n text-decoration: none;\\\\n padding-top: 0px;\\\\n color: #2970A6;\\\\n}\\\\na:link {\\\\n color: #2970A6;\\\\n text-decoration: none;\\\\n}\\\\na:hover {\\\\n color: #2970A6;\\\\n text-decoration: underline;\\\\n}\\\\np.buttonlink {\\\\n margin-bottom: 24px;\\\\n}\\\\n.copyright {\\\\n font-size: 12px;\\\\n color: #666666;\\\\n margin: 5px 5px 0px 30px;\\\\n}\\\\n.copyright_alone {\\\\n font-size: 12px;\\\\n color: #666666;\\\\n margin: 35px 5px 0px 30px;\\\\n}\\\\n.details {\\\\n font-size: 14px;\\\\n color: #969696;\\\\n border: none;\\\\n padding: 20px 20px 20px 20px;\\\\n margin: 0px 10px 10px 35px;\\\\n}\\\\n\\\\n.shadow {\\\\n border: 3px solid #9f9f9f;\\\\n padding: 10px 25px 10px 25px;\\\\n margin: 10px 35px 0px 30px;\\\\n background-color: #ffffff;\\\\n width: 600px;\\\\n\\\\n -moz-box-shadow: 3px 3px 3px #cccccc;\\\\n -webkit-box-shadow: 3px 3px 3px #cccccc;\\\\n box-shadow: 3px 3px 3px #cccccc;\\\\n /* For IE 8 */\\\\n -ms-filter: \\\\\\\"progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc')\\\\\\\";\\\\n /* For IE 5.5 - 7 */\\\\n filter: progid:DXImageTransform.Microsoft.Shadow(Strength=5, Direction=135, Color='cccccc');\\\\n}\\\\n.logo {\\\\n border: none;\\\\n margin: 5px 5px 0px 30px;\\\\n}\\\\n</style>\\\\n\\\\n</head>\\\\n\\\\n<body>\\\\n<div class=\\\\\\\"logo\\\\\\\"></div><p>&nbsp;</p>\\\\n<div class=\\\\\\\"shadow\\\\\\\">\\\\n<h1>Page Redirection</h1>\\\\n\\\\n\\\\n<p>\\\\nThis request is being redirected. If this page does not automatically\\\\nredirect, <a href=\\\\\\\"https://*****PROXY****-proxy/B0001D0000N0001N0001F0000S0000R0004/10.203.168.180/https://10.203.168.180:9096/\\\\\\\">click here to proceed</a>&nbsp;.\\\\n</p>\\\\n\\\\n</div>\\\\n\\\\n\\\\n</body>\\\\n</html>\\\\n\\\"\""

So apparently loki is going over the proxy to address the grpc port.

Can you give advice how this should be configured?

Luc