I’m using grafana
+prometheus
+telegraf
for monitoring certain web servers. I’m collecting metrics for certificate information: x509_cert_expiry
and dns status: dns_query_rcode_value
exposed by telegraf
for a single web server, say web1
, where two websites are hosted, say http://www.myapp1.com
and http://www.myapp2.com
. Metrics look like this:
x509_cert_expiry{common_name="myapp1.com",env="staging",group="web",host="...",instance="...",issuer_common_name="...",job="web1",public_key_algorithm="RSA",san="myapp.com",serial_number="...",signature_algorithm="SHA256-RSA",source="https://www.myapp1.com:443",verification="valid"} 4898676
x509_cert_expiry{common_name="myapp2.com",env="staging",group="web",host="...",instance="...",issuer_common_name="...",job="web1",public_key_algorithm="RSA",san="myapp.com",serial_number="...",signature_algorithm="SHA256-RSA",source="https://www.myapp2.com:443",verification="valid"} 4897854
dns_query_rcode_value{domain="myapp1.com",env="staging",group="web",host="...",instance="...",job="web1",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
dns_query_rcode_value{domain="myapp2.com",env="staging",group="web",host="...",instance="...",job="web1",rcode="NOERROR",record_type="NS",result="success",server="8.8.8.8"} 0
(values containing ‘…’ are redacted - not an actual label values)
Now, I would like to combine/join/merge/squeze these two metrics x509_cert_expiry
and dns_query_rcode_value
into a single grafana
table just to avoid creating two panels one for certificate information and other for domain probe, in order to display everything in a single panel since these two metrics share some common information (e.g. common_name="myapp1.com"
→ domain="myapp1.com"
) and are scattered across multiple metrics with different labels.
Looking at the collected metrics, there are some similar label values and I would like to use to produce single set of data with specific columns - some of them pointing to label values, some of them having values from metrics.
What I’ve tried so far:
- creating
promql
expression for joining these two vectors:dns_query_rcode_value * on (instance) group_left(san) (count by (instance, san) (x509_cert_expiry{san!=""}))
but that isn’t working:Error executing query: found duplicate series for the match group {instance="..."} on the right hand-side of the operation: [{instance="...", san="..."}, {instance="...", san="..."}];many-to-many matching not allowed: matching labels must be unique on one side
- Messing with
grafana
transforms as per Merge/join two metrics in Prometheus/PromQL - Stack Overflow and Merge/join two metrics in Prometheus/PromQL - Stack Overflow but no avail.
Is something like this even possible ?
Thanks.