Mixed query with prometheus

Grafana Cloudwatch
1

Dear all,

I’m trying to create a mixed query between cloudwatch (S3) and a custom metric I’m sending to prometheus. My goal is to add all bucket tags (sent by my custom metric) to the bucket info sent by cloudwatch . I’m not sure if I’m following the right approach so, if anyone already got this working, happy to learn his/her way.

The mixed query itself work fine and I endup with a “table” with prometheus info and the info from cloudwatch, but when I want to join based on a field “bucket” (bucketname) I cannot do that becasue my “bucket” filed from prometheus is the real bucket name but the bucket value from cloudwtach is the “bucketname storagetype” .
I did not find a way to remove the storage type from the value and I do not know how to proceed (lack of ideas), so, first question woule be if is there a way to remove a string from a field (not value)?
this is how this looks like (notice the “StandardSotrgae” next to the bucket name.

image
image586×112 9.73 KB

if this is not possible, any idea on how to “join” the S3 tags information to the cloudwatch S3 info?

Thanks in advance

2

Provide reproducible example, pls

3

sorry, a bit new here. what do you exactly mean and, most important, how?

4

How one person can replicate your case? (Grafana version, queries, transformations, …). I guess it can be done on the query level, but you didn’t show any query. Provide queries in text format, not just screenshots, so when someone want to help then he must rewrite query from the image into text into simple copy&paste.

SO has good guide how to ask: How do I ask a good question? - Help Center - Stack Overflow

5

thanks, I need some time and I’ll come back with an example.

6

So, it’s a mixed query between prometheus (custom Gauge) and Cloudwatch (S3)

Prometheus looks like this (not sure if this is enough as I did not find a better way to share the query

“schema”: {
“refId”: “xchange_status”,
“meta”: {
“type”: “numeric-multi”,
“typeVersion”: [
0,
1
],
“custom”: {
“resultType”: “vector”
},
“executedQueryString”: “Expr: xchange_bucket\nStep: 2m0s”,
“preferredVisualisationType”: “rawPrometheus”
},
“fields”: [
{
“name”: “Time”,
“type”: “time”,
“config”: {
“interval”: 120000
}
},
{
“name”: “name”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “BucketName”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “expiration_date”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “expired”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “instance”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “job”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “legacy”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “requestor”,
“config”: {
“filterable”: true
},
“type”: “string”
},
{
“name”: “Value”,
“type”: “number”,
“config”: {}
}
]
}

and produces output like this (CSV) (single row):

Time, name, BucketName, expiration_date, expired, instance, job, legacy, requestor, Value
33:41.2, xchange_bucket, bi-bio-name-xchange, 2024-01-01T00:00:00Z, TRUE, servername.eu.domain.com, integrations/unix, 1, name@domain.com, -438

so, with this first query I get all Bucket tags, and I’m interested in BucketName and expired. This is the info I’d like to “join” to my second query.

The second query is cloudwatch and looks like this:

REMOVE_EMPTY(SEARCH(‘{“AWS/S3”,“BucketName”,“StorageType”} MetricName=“BucketSizeBytes” :aws.AccountId=“512XXXXX5801”’, ‘Average’, 10800))

and produces output like this (CSV):

Time, {“bucket”:“51XXXX01-terraform-state-bucket StandardStorage”}
15/03/2025 23:38, 161021

In some case, when the bucket is stored in 2 different sotrgae types, I get entries like

> Time, {bucket:51XXXX01-terraform-state-bucket StandardStorage}
15/03/2025 23:38, 161021
> Time, {bucket:51XXXX01-terraform-state-bucket StandardIAStorage}
— —
15/03/2025 23:38, 317021

So, for the same bucket I get two entries based on the Storage type.
(this is just ane example)

I’d say transformations themselves are not that relevant as I only have “labels to fields” to get a “BucketName” filed; something like:

Time, 51XXXX01-terraform-state-bucket, BucketName, Series
15/03/2025 23:38 , 161021, 51XXXX01-terraform-state-bucket, bi-bio-abiosciences-xchange StandardIAStorage

Now, if I go for the join based on BucketName, I get something like this (and sorry here cause I do not really know how to show this aprt from this screenshot:

image
and continues like this:
image

so, as you can see, the merge works, but the value is still the value from the first query and not the size of each bucket in each storage type. The table itself become really complex to manage and now merging values from the two fields it’s almost impossible as every bucket has the value under a differnet column name.

Not sure if I’m doing something wrong or that my approach is, simply, wrong (based on the result and the complexity I see on the produced table) and there’s a simplest wy to embbed this “expired” tag information to the bucket in a different way.

thanks

7

I ended up creating a custom metric that reports both the bucket tags and the bucket size.
I did not find a way to merge the information from both tables.