Service Account API log submission error

Grafana Loki
1

I have a new Grafana Cloud account, to which I am attempting to send test logs via Loki. My understanding is that using a Service Account for programmatic log submission is the current best practice for authentication. I have created a Service Account with “API Keys:Writer” permission, and a token for that service account. I am attempting to submit test material to my account with the following bash script:

#!/bin/bash

GRAFANA_CLOUD_INSTANCE_ID="<REDACTED>"
GRAFANA_CLOUD_TOKEN="<REDACTED>"
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

curl --verbose \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${GRAFANA_CLOUD_TOKEN}" \
  --data-raw "{\"streams\": [{\"stream\": {\"app\": \"foo\"}, \"values\": [[\"${TIMESTAMP}\", \"dummy message\"]]}]}" \
  "https://logs-prod-${GRAFANA_CLOUD_INSTANCE_ID}.grafana.net/loki/api/v1/push"

I have attempted this script with the following variations:

  • with my Grafana Cloud instance ID in the URL (as above) - {"code":"Error","message":"It looks like there is an issue with this instance."}
  • with the 3-digit string in my Loki data source configuration in the URL (rather than my instance ID) - {"status":"error","error":"authentication error: legacy auth cannot be upgraded because the host is not found"}
  • without port 3100 in the URL (as above)
  • with port 3100 in the URL (timeout)

It is unclear to me what I might be doing incorrectly, and neither the documentation, nor Google, nor ChatGPT have provided any guidance outside of the strategies outlined above. Any insight or direction for further troubleshooting would be much appreciated.

2

You have section “Send Logs” in your Grafana Cloud Admin UI and there are examples, e.g.

image
image704×478 16.5 KB

As you see those examples are using basic auth not bearer auth, so this is not correct:

-H "Authorization: Bearer ${GRAFANA_CLOUD_TOKEN}" \

Basic auth can be defined for example as (check your curl doc for more details and fix any syntax issues):

-u "${GRAFANA_CLOUD_USERNAME}:${GRAFANA_CLOUD_TOKEN}" \
3

This is a question about how to use Service Accounts. I understand that there are other ways to authenticate. I’m not using e.g. Grafana Alloy (the example in your screenshot) to post logs.

4

Yes, I understand. But use common sense - if Alloy example is using basic auth, then why you don’t use basic auth?

I gave you also an example how to use basic auth with curl

-u "${GRAFANA_CLOUD_USERNAME}:${GRAFANA_CLOUD_TOKEN}" \

That “-u” is an example how to use basic auth with curl:

% curl -h
Usage: curl [options...] <url>
...
 -u, --user <user:password> Server user and password

Be familiar with terms “basic auth” (what Grafana cloud is using) vs “bearer auth” (what you used in your script, which is not correct).

5

I don’t think that I have a common sense issue. You seem to think that this problem should be solved with Basic Auth. You’re entitled to your opinion, but I’m trying to solve this problem with a Service Account. The curl examples in the documentation for Service Accounts use Bearer tokens:

If you don’t want to assist me in answering the question that I’m asking, please feel free to stop participating in this thread. Alternatively, you could provide me with an explanation as to why you feel that Service Accounts are the wrong approach.

6

OK, you should be explicit, which documentation you are using in the first place.
Service account in this case is to access “Grafana” (that is just app, which is reading data from Grafana Cloud or anywhere else).
But you are pushing logs into “Grafana Cloud”, not into “Grafana”. So you can’t use service account from Grafana to authenticate against Grafana Cloud.

Grafana Cloud - access policies:

Grafana - service accounts: