Statsd to graphite with Hosted Metrics

We are trialing Hosted Metrics. Our applications send metrics to statsd, which then sends those metrics to a Graphite server. The only requirement to do this for our self-managed Graphite server is to put the correct FQDN as the value for the graphiteHost property in the config.js for statsd.

I do not see any explicit information for how to integrate statsd with Hosted Metrics. Our Hosted Metrics org page lists the following as our “Metrics Endpoint”, which is “The URL you use to send data to us”: https://tsdb-<our_org_unique_subdomain>.hosted-metrics.grafana.net/metrics. To this end, I took a guess that we could do

  graphiteHost: 'tsdb-<our_org_unique_subdomain>.hosted-metrics.grafana.net',

in our statsd config.js. I am not seeing our metrics appear in our Hosted Grafana, however.

The Hosted Metrics page claims

GrafanaCloud includes a high performance metrics backend, fully compatible with Graphite and Prometheus.

We would appreciate help in connecting our statsd to Hosted Metrics’ Graphite-compatible backend.

Hi @clasherhisc, opening your ticket now in our support system. :slight_smile:

I wish you would have gave the answer here in the forums where others could see how to solve this issue. I’m having the same troubles.

@kennethcassel It’s unfortunate this was not captured here nor replied to by a Grafana Cloud representative, but I will share with you what we found.

You cannot directly send metrics from statsd to Hosted Metrics. This is by design of Hosted Metrics, because the statsd protocol is inherently insecure, as it does not support authentication. You will instead need to send your traffic from statsd to a carbon-relay-ng daemon, which can provide authentication in the form of your Hosted Metrics API key. The carbon-relay-ng daemon will forward the metrics on to Hosted Metrics. So your overall flow of metrics is:

your application → statsd → carbon-relay-ng → Hosted Metrics

In your carbon-relay-ng.conf file, you will want a section that looks similar to the contents below:

# Routes
# See https://github.com/graphite-ng/carbon-relay-ng/blob/master/docs/config.md#Routes
[[route]]
key = 'grafanaNet'
type = 'grafanaNet'
addr = 'tsdb-<your_org_unique_subdomain>.hosted-metrics.grafana.net'
apikey = '<your_hosted_metrics_api_key>'
schemasFile = '/etc/carbon-relay-ng/storage-schemas.conf'

Note the content of your storage-schemas.conf file have no impact whatsoever on the storing and aggregation of metrics in Hosted Metrics, but we found it necessary to have an existing file. Here is an example, but again, the values for retentions don’t matter:

[default]
pattern = .*
retentions = 10s:8d,1m:60d,30m:2y

I hope this helps you and your team.

Thanks for the reply!

I ended up just writing a different statsD backend that allowed posts to Hosted Metrics. statsD supports different backends, so I created one that accepts an API Key and endpoint to post to.

We got carbon-relay-ng working to send metrics as well.

Not sure which way we will go because we wanted to use a hosted solution to reduce complexity and maintenance for our team. If we have to manage another machine with carbon-relay-ng, it’s an extra step that we don’t want to do unless necessary.