Grafana + AWS S3

Hi,

I am trying to use CloudWatch as data source and get the data from AWS S3 to build charts & graphs in Grafana. I am struggling to get this dashboard started.

I have created a free AWS account and created bucket in AWS S3 and uploaded some test data using json and CSV files to buckets.

I have added Cloud watch as datasource in Grafana using AWS access key and secret key. Now I am not 100% sure how to build the dashboard from the data available in AWS S3 buckets. I am referring to this , but can’t really get to how to build charts from data available in buckets !!

Can someone assist on this !

I don’t think you can use Grafana to visualize CSV or JSON files that are stored in S3. The CloudWatch integration allows you to visualize metrics available through CloudWatch, i.e. the performance metrics of the S3 service itself (which by the sound of it is not what you want?).

To visualize the data you’ll need to put it in a database like MySQL or InfluxDB - then connect that as the Grafana data source. Hope that makes sense?

thanks for the reply.

So, it’s not possible to visualize the data available in AWS S3 buckets through Grafana. We can only monitor S3 services performance through Grafana. Is that right ? @svetb

Yes, basically. Some workarounds come to mind, but they’re not “nice”.

One option may be to install the Simple JSON Datasource (https://github.com/grafana/simple-json-datasource) and set up your file in S3 so that a request to https://your_s3_url/query returns a JSON payload that that plugin can parse (see the link for an example). Though that may require some major rejigging of your JSON.

The load-from-CSV option is discussed here: CSV file as a datasource?

thanks for the info and suggestions

You can use AWS Athena (it is a data warehouse solution, so don’t expect performance of standard TSDB) and visualize Athena results with https://github.com/mtanda/grafana-aws-athena-datasource

2 Likes

You can use S3 Select API. I have created a plugin for the same


Requires Grafana 7.
1 Like

Can this be installed on Grafana cloud (starter edition)? I’d like to try it out

If it supports installing backend plugins then it can be installed. I haven’t tried Grafana Cloud so not sure