How to expose prometheus metrics on instance level

[mrlight]

Hi,
I have a plugin which runs under multiple instances in the grafana env. I want to expose some prometheus metrics on instance level. Eg: I would like to reuse the *sql.db.stats in a custom collector by calling the collect function. This is working like a charm if only one instance of the plugin is running. When i have a second one, I receive “duplicate metrics collector registration attempted” errors. Is there a way to get a registry by instance? with its own endpoint? like:metrics/plugins/myplugin/myinstance? or how to handle this situation?

[anon34853223]

Hey. I’m not sure I understand what “reuse the *sql.db.stats in a custom collector by calling the collect function” means.

Given you have a backend plugin and you want to register metrics, either register the metrics when plugin starts, e.g. grafana-plugin-examples/examples/datasource-basic/pkg/main.go at main · grafana/grafana-plugin-examples · GitHub, or either use sync.Once in datasource instance to guarantee the metrics are only registered once. To track per instance you would have to use metric labels.

Let me know if I completely misunderstood the question and/or provide more details for us to be able to understand and help.

[mrlight]

Thanks for your detailed reply. It helps a bit but probably my problem is more prometheus generic. I’m not really an expert on that.
What would I like to do: I would like to generate metrics for my datasource plugin. There is the possibility to use the gauge or counter interfaces (with promauto) like described in the grafana-doc.
On the other hand I have internal counters or state objects from subsequent libraries. There I thought it would be possible to just copy the actual state of this object during the prometheus collect. But for that I need a reference to that object in my collect function which seems not to be possible with a global collect and multiple instances.

type LocalPrometheusCollector struct {
	db                        *sql.DB
	setting                   *backend.DataSourceInstanceSettings
	db_in_use_connections     *prometheus.Desc
	db_wait_count             *prometheus.Desc
}
func (c *LocalPrometheusCollector) Describe(ch chan<- *prometheus.Desc) {
	ch <- c.db_in_use_connections
	ch <- c.db_wait_count
}

func (c *LocalPrometheusCollector) Collect(ch chan<- prometheus.Metric) {
	ch <- prometheus.MustNewConstMetric(c.db_in_use_connections, prometheus.GaugeValue, float64(c.db.Stats().InUse), c.setting.Name)
    ch <- prometheus.MustNewConstMetric(c.db_wait_count, prometheus.CounterValue, float64(c.db.Stats().WaitCount), c.setting.Name)
}

func NewLocalPrometheusCollector(db *sql.DB, setting *backend.DataSourceInstanceSettings) *LocalPrometheusCollector {
	return &LocalPrometheusCollector{
		db:                        db,
		setting:                   setting,
		db_in_use_connections:     prometheus.NewDesc("grafana_plugin_sql_pool_in_use_connections", "SQL Pool - The number of connections currently in use.", []string{"instance_name"}, nil),
        db_wait_count:             prometheus.NewDesc("grafana_plugin_sql_pool_count_total", "SQL Pool - Connections total wait count", []string{"instance_name"}, nil),
}

//in:
func newDataSourceInstance(ctx context.Context, setting backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
...
prom := NewLocalPrometheusCollector(db, &setting)
prometheus.DefaultRegisterer.MustRegister(prom)
...

With that I will receive a

duplicate metrics collector registration attempted

by shifting the mustRegister() into main i will not be able to access the objects on instance level.

All my googeling brougth me to the solution to register endpoints per instance…

In the example above the problem is the counter metric. I haven’t found a way to use the library based counter by using the promauto approach. Since you have to inc it somehow by having a library counter value only. (got it?)

[anon34853223]

Any available library for tracking db stats seems to support only one database, e.g. GitHub - dlmiddlecote/sqlstats: A Go library for collecting sql.DBStats in Prometheus format. I’m afraid we cannot help with this specific use case with using custom collectors, but can provide some pointers/ideas.

by shifting the mustRegister() into main i will not be able to access the objects on instance level.

You can probably work around it, but you’re on your own and would need to manage concurrency/locks when new instances are created and old ones disposed.

In general, if your collector is available to each instance you could add/remove your instance with the collector, something like

func newDataSourceInstance(collector *LocalPrometheusCollector) datasource.InstanceFactoryFunc {
  return datasource.InstanceFactoryFunc(func(ctx context.Context, setting backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
    ...
    instance := &MyDatasource{
      collector: collector,
    }
    collector.Add("<name>", <instance/db/stats>)
    ...
  })
}

func (d *MyDatasource) Dispose() {
  d.collector.Remove("<name>")
}

// main
prom := NewLocalPrometheusCollector()
prometheus.DefaultRegisterer.MustRegister(prom)
err := datasource.Manage("myds-plugin-id", newDataSourceInstance(prom), datasource.ManageOpts{})
...