Datasource Index Patterns

Grafana Elasticsearch
1

Hi everyone,

I am trying to get two differently named ES indices in the same datasource but I’m having trouble because they both use date patterns and have very similar, but slightly different names.

example
index pattern 1:
[metrics_for_infra_net_company_]YYYY.MM.DD
index pattern 2:
[metrics_for_infra_esx_company_]YYYY.MM.DD

context:
The infra_net index is getting too big and we are switching part of the agents from that datasource to a new one (esx) to alleviate the load on the first index. The datasource for the new index (esx) does not contain historical data and I would like to temporarily have it read both indices in order to avoid having to modify every panel on our dashboards to switch the datasources to mixed mode and copy /paste the queries.

Ideally, I’d like to be able to do something like this:
datasource esx(temporary)
[metrics_for_infra_[esx|net]_company]YYYY.MM.DD

The grafana documentation is very limited on index pattern info for datasources. Maybe I missed something and you guys can help me achieve my dream DataSource?

Solutions I am aware of
Using * in the datasource:
[metrics_for_infra_*_company]YYYY.MM.DD

This would work, but unfortunately I have other equally large indices that follow this exact pattern and I fear the request would be problematic for our ES cluster.

Using aliases in ElasticSearch:
Since new indices are created daily, I would have to put the alias in the templates, but I would lose the whole YYYY.MM.DD optimization in Grafana.

I hope my question was clear enough to understand, and I thank you for taking the time to read this!

2

Maybe create an alias for each day to be able to use the pattern in Grafana.

3

This needs to be scripted to be manageable but I guess it is the cleanest solution.

Are there any plans to include regexp-like functionality in the “index name” field ?

Thank you.

4

There’s no plans for supporting that and no open feature request. Does Kibana support this?

5

This is indeed not possible in Kibana while creating an index pattern. That doesn’t mean it shouldn’t be implemented in Grafana :smiley: .

As far as I understand it should be doable by requesting the list of indices (GET _cat/indices) and then parsing the results with the provided regexp (although on large clusters this could be a little slow…).

I don’t know how the check mechanism works in Grafana when you hit Save & Test for a pattern so I really have no idea if it’s feasible.

As a sidenote, Kibana seems to be dropping support for the YYYY.MM.DD auto expansion when creating an index pattern.

Expand index pattern when searching [DEPRECATED]
With this option selected, searches against any time-based index pattern that contains a wildcard will automatically be expanded to query only the indices that contain data within the currently selected time range.
Searching against the index pattern logstash-* will actually query Elasticsearch for the specific matching indices (e.g. logstash-2015.12.21 ) that fall within the current time range.
With recent changes to Elasticsearch, this option should no longer be necessary and will likely be removed in future versions of Kibana.

This text is from a Kibana in v5.6.8. when creating a new index pattern. In Kibana 6.4.2 the option no longer exists.

1 Like