I’m trying to setup Grafana Cloud so that I can instrument a NodeJS app and Postgres database. However, I’m seeing sporadic outages (see attached screenshot) where I seem to be getting 403/401 errors back from the Prometheus and Loki endpoints. Running journalctl -u grafana-agent -e on the server, I see a lot of this:
May 25 22:43:25 production grafana-agent[152589]: ts=2023-05-25T22:43:25.302111299Z caller=dedupe.go:112 agent=prometheus instance=ac401d6107d4cb4ec388037555fb9bce component=remote level=error remote_name=ac401d-bc4a0a url=https://prometheus-us-central1.grafana.net/api/prom/push msg="non-recoverable error" count=7 exemplarCount=0 err="server returned HTTP status 403 Forbidden: "
I’ve tried multiple different Service Accounts/Service Tokens, API Keys, and Access Policies. So far the only thing that has (kind of) worked is an API Key that I created through my account page on Grafana Cloud, but I’m still seeing the intermittent 403 errors. I’m also seeing a lot of messaging around API Keys being deprecated, so I’m surprised that none of the alternatives have worked at all for me. Any ideas on what could be causing this?
- Grafana v10.0.0-cloud.2.aef9c3bd (9e6ab163ca)
- Server OS: Ubuntu 22.04.2 LTS
- I installed and setup the agent by following the instructions for the NodeJS connector
