Too many 500 Errors occurs from Grafana sidecar

Hi, I just deployed kube-prometheus-stack Helm Chart and use Grafana values in default, but it seems to be an error in kiwigrid/k8s-sidecar.

Logs printed like below:

at 0x7f3078655e90>: Failed to establish a new connection: [Errno 111] Connection refused')': /api/admin/provisioning/dashboards/reload"}
{"time": "2023-09-10T16:49:08.469894+00:00", "level": "ERROR", "msg": "Received unknown exception: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/admin/provisioning/dashboards/reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3078656f90>: Failed to establish a new connection: [Errno 111] Connection refused'))\n"}
Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/app/.venv/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f3078656f90>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/.venv/lib/python3.11/site-packages/requests/adapters.py", line 489, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 815, in urlopen
    return self.urlopen(
           ^^^^^^^^^^^^^
  [Previous line repeated 2 more times]
  File "/app/.venv/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/admin/provisioning/dashboards/reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3078656f90>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/resources.py", line 347, in _watch_resource_loop
    _watch_resource_iterator(*args)
  File "/app/resources.py", line 335, in _watch_resource_iterator
    request(request_url, request_method, enable_5xx, request_payload)
  File "/app/helpers.py", line 132, in request
    res = r.post("%s" % url, auth=auth, json=payload, timeout=REQ_TIMEOUT)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.11/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=3000): Max retries exceeded with url: /api/admin/provisioning/dashboards/reload (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f3078656f90>: Failed to establish a new connection: [Errno 111] Connection refused'))

What is the problem and how should I act?

The error you’re encountering indicates a connection problem between kiwigrid/k8s-sidecar and Grafana’s API running on localhost:3000. First, ensure Grafana is running and accessible.

Review configurations for both services, checking URLs and ports. Investigate network policies and firewall settings in your Kubernetes cluster that may block communication.

Confirm all necessary dependencies are correctly installed. Examine logs for error messages. Verify both kiwigrid/k8s-sidecar and Grafana are deployed in the same namespace. Avoid port conflicts.

Finally, check environment variables and configuration files for potential issues. Resolving these factors should help establish a successful connection between the services.

I am getting the same error after upgrading the kube-prometheus-stack to 5.55.1 , @kworkbee were you able to find the solution please?

1 Like

Im having the same issue running 57.2.0 helm chart version