How to push Prometheus metrics in Grafana Cloud?

[pierrelannoy]

Hello all!
First of all, I would like to apologize if this question is trivial - or already answered many times - but I must confess that I am a bit lost…

I am developing an application that is able to send its own metrics or publish an endpoint ready to be scrapped by Prometheus. This works perfectly when pushing to a Pushgateway - so without agent.

Now I wish I had the same way of working with Grafana Cloud and sending those metrics there, again, without agent. I understand that an endpoint is available for this (https://<host>.grafana.net/api/prom/push). But no matter how I do it, it doesn’t work (HTTP 400 or 404).

Could you tell me if I can actually use this endpoint as if it is a pushgateway? If so, how can I do it? If not, what is the “recommended” way to send metrics?

Many thanks in advance for your help

[ximenaaliaguilla]

Hi Pierre!

The recommended way of sending your metrics to a Grafana Cloud Prometheus endpoint is not through a push gateway but via remote_write from either a local Prometheus or using the Grafana Agent, via their config YAML file. Since you’ve already developed a way to scrape your application’s metrics to a (local) Prometheus then this is likely the easier path for you than the agent. You’ll add the remote_write to your local Prometheus’ yaml config as shown here: Prometheus | Grafana Labs

Wherein:
url =
https://prometheus-blocks-prod-us-central1.grafana.net/api/prom/push (confirm this in your Grafana Cloud Portal under Prometheus > Details as some accounts may vary in their URL path)
username = Prometheus instanceID (also found in your Grafana Cloud Portal under Prometheus > Details)
password = Grafana.com API key (generated from your Grafana Cloud portal under “API Keys”)

To access your Grafana Cloud portal, go to Grafana.com > “My Account” or “Login”. Hope this was helpful!

[pierrelannoy]

Hello @ximenaaliaguilla and thanks for your answer.

Unfortunately, the production version of my app will not be public - so no way to scrap from Grafana Cloud (it’s only working on my dev environment to test the prometheus rendering).

I had hopes of doing something that didn’t require a local prometheus, like if my app is its own agent and uses directly the remote write endpoint…

[fxpester]

I’m trying to do the same: “send metric to grafana cloud using only curl”
did you succeed ?

[ximenaaliaguilla]

Tip: Be sure that your Grafana.com API key has the “MetricsPublisher” type role.

[ximenaaliaguilla]

@fxpester the only fully supported way to send metrics to a Grafana Cloud hosted metrics endpoints are via the remote_write mechanism I described in an earlier reply (from an external Prometheus or the Grafana Agent) or, for Graphite metrics, as described here.

There is another experimental way to push Telegraf metrics to the hosted Prometheus endpoint in Grafana Cloud that you’re welcome to try here: Store and visualize Influx Telegraf data | Grafana Labs

However, using curl to push metrics to a Grafana Cloud hosted metrics endpoint would not otherwise be supported.

[akauppi]

I understand that this is currently (Jul 2022) not available. However, to bring Grafana’s attention to why people would try to feed it “using only curl”, here’s my use case:

I’d like to collect metrics from web app front-end and show them in Grafana Cloud.

The plan is to proxy these via a Firebase backend (I’d only collect metrics from logged in users). Cloud Functions and maybe PubSub can gather small droplets of metrics together and deliver them either when scraped, or scheduled.

I would use the prom-client Node.js client, likely, but there’s not going to be a Prometheus instance running all the time (that would defeat the Grafana Cloud’s SaaS aspect, right?)

[daviddorman]

@akauppi depending on your current setup, you could send data directly from your application using our newly released support for ingesting Influx Line protocol into Grafana Cloud. You can now point your Telegraf and other services pushing Line protocol metrics to GrafanaCloud via HTTP.

Specifically, here’s how you would push from applications directly.

Let me know if this is helpful or if you have any questions.

[alexsey]

I have tried to implement the method in the docs, and it’s responding to me with 400 “bad metrics write request\n”. There are a bunch of issues with the doc:

1. At the start of the article, there is a link to Cloud Protal, but it is linking the self page i.e., the page of the doc
2. In Pushing from applications directrly although the text is saying “use the same endpoint as above but with…” the example URL is identical to the one above in the article: " https://influx-prod-03-prod-us-central-0.grafana.net/api/v1/push/influx/write". I assume it should be different, unless this section is giving no new information. And there definitely should be no “but with” if it expected to be identical

My setup:

• I have only default data sources
• I’ve taken username and url from Prometheus data source
• For the password, I’ve used a token generated at https://grafana.com/orgs/<ORG_NAME>/api-keys
• The initial URL was https://prometheus-prod-10-prod-us-central-0.grafana.net/api/prom
• Converted URL became https://influx-prod-10-prod-us-central-0.grafana.net/api/v1/push/influx/write
• The request is POST with
• The request body is
  test,test_label=test_label_value direct_metric=22 1661358881969

What can you recommend to try to fix this?

Thank you

UPD: ok, there is an important bug in the dosc: all values MUST be quoted. Example
diskio,host=work,name=dm-1 write_bytes=651264i,read_time=29i 1612356006000
is incorrect - values must be
diskio,host=work,name=dm-1 write_bytes="651264i",read_time="29i" 1612356006000
With such a fix, the response is 204.
Although I still cannot see the metric in the Explorer, so the problem is not jet resolved
UPD2: so here is the minimal example to reproduce - the curl that is returning 204, yet no data appears in the Explorer
curl -d 'test,test_label=test_label_value direct_metric="22" 1661368910216' 'https://<USER>:<TOKEN>@influx-prod-10-prod-us-central-0.grafana.net/api/v1/push/influx/write' -i

[johnmarbach2]

Hey alexsey,

John from Grafana Labs here. Sorry for the trouble with the InfluxDB Line Protocol.

A few tips that should help you get to a solution:

1. The values do not need to be encapsulated with quotes.
2. If specified in your data payload, the timestamp value must be encoded in Nanoseconds.
3. You may also push metrics without specifying a timestamp. We’ll use the timestamp for when we received your request.

For example, please try the following two payloads and then visit Explore:
foo_metric,bar_label=abc cpu_load=55.3

foo_metric,bar_label=abc cpu_load=74.1 1661369185000000000

The example timestamp you’re using, 1661358881969, is in Milliseconds. I verified that’s what our docs are also showing – timestamp examples in Milliseconds. Sorry for the confusion. We’ll update our docs shortly to reflect the proper timestamp values.

[alexsey]

It works, thank you! At some point, I’ve tried not to use timestamp at all, but at that moment, my values were already quoted %) Interesting how quoting some point allowed me to move from 400 to 204… anyway, I really appreciate your help and the time it takes for you to give the answer! When you will be fixing the docs, it may be also helpful to some folks to note that the password from the Prometheus data source will not work, and the API key from https://grafana.com/orgs/<ORG_NAME>/api-keys should be used instead. The first time I hit this problem with Loki it took a few hours to understand how to set it up properly

[johnmarbach2]

Noted. Thank you for your feedback!

[akauppi]

I also got pushing metrics via Influx line protocol working. But… now what? Where can I see the dummies I’ve posted?

The same applies to Loki. I have a curl command that seems to write (gives 204), but where can I see the logs?

curl -v -XPOST -H "Content-Type: application/json" -s "$(_LOKI_URL)/loki/api/v1/push" --data-raw \
 '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ $(_NOW_NANO), "fizz buzz" ] ] }]}'

[yosiasz]

for influx go to

ip.of.influx:8086

for loki

http://ip.of.loki:3100/metrics

[akauppi]

While that may be correct in the general context, this message is in the forum for Grafana Cloud.

Expecting:

• to see my dummy metrics (and logs) in a Grafana Cloud UI

[yosiasz]

please post your sample influx data as inline csv

name,time,host,topic,value
mqtt_consumer_float,1662507929695650760,vps-362a4be9,GPS/Latitude,43.5785
mqtt_consumer_float,1662507940992628730,vps-362a4be9,GPS/Longitude,1.44712
mqtt_consumer_float,1662508438504964767,vps-362a4be9,GPS/Latitude,43.5784
mqtt_consumer_float,1662508449814560959,vps-362a4be9,GPS/Longitude,1.44725
mqtt_consumer_float,1662508934276570198,vps-362a4be9,GPS/Latitude,43.5786

Then you can plot it using flux query.

[akauppi]

Thanks for the screenshot.

Some clarifying questions.

Setup:

• Grafana Cloud free tier, i.e. I have the predefined stack (picture below).

After having posted Influx metrics and Loki logs (presumably successful, since I got 204’s), I still see this:

Expected:

The buttons should have changed from “Send Metrics/Logs” to something else, since there now is data in the system.

Actual:

It seems to me that the GUI is not aware of the data.

[akauppi]

@daviddorman Should Influx Line protocol work also for Grafana Cloud free tier?

[daviddorman]

@akauppi yes, it should work regardless of plan.

This part of the UI should acknowledge if you are successfully sending data under ‘Current Active Series’ (albeit the button will always just say 'Send Metrics - we will fix that in the future).

Send a note to cloud-success@grafana.com so we can work with you on this and reference this thread.

[vchernetskyi993]

Hello folks,

Is there a possibility to send batch messages to the same /influx/write endpoint?

Per ILP docs (Line Protocol | InfluxData Documentation Archive), \n should be used as a separator. But when I try:
curl \ -d 'test,test_label=test_label_value direct_metric=22\ntest_1,test_label_1=test_label_value direct_metric=23' \ -i \ -u <username>:<key> https://influx-prod-01-eu-west-0.grafana.net/api/v1/push/influx/write

I receive 400 status and “error parsing points” message.

P.S. I can send 2 separate requests, but the application has a lot of different metrics. And sending them 1 by 1 seems not an effective solution.