Grafana Native S3 Integration through AWS IAM Role

  • What Grafana version and what operating system are you using?
    Using Grafana Cloud v9.5.7 do not have available details around OS usage, assume a entreprise version of Linux.

  • What are you trying to achieve?
    Retrieve and render S3 stored data using AWS IAM Role authentication scheme.

  • How are you trying to achieve it?
    To our knowledge Grafana does not natively support this feature, it does not allow to retrieve a JSON or CSV file stored in a S3 Bucket and use that content to render some dashboard panels. The team established a PoC with the usage of a open source plugin yesoreyeram-infinity-datasource which does include an AWS integration method through long-term IAM User credentials. As a side note this is a temporary measure seeing this is expected to migrate to our primary data source SnowFlake by EoY.

  • What happened?
    Once moving the PoC to our live environments, a security concern was raised around the AWS authentication method utilized in said plugin prompting a challenge to instead utilize a more organizational streamline AWS authentication method, through a specific IAM Role.

  • Did you follow any online instructions? If so, what is the URL?
    To prepare the PoC the team has based their development from the following article AWS API | Grafana Plugins documentation

Try dofferent approach - Athena

You may also try to access s3 data via http (and have IAM in a place, e. g. s3 bucket policy which allow only Grafana role to have access)

Indeed Athena is a perfectly viable option, if not for this solution representing a temporary measure…
The idea would be at the earliest convinience to migrate this data source to a more structured and flexible design, in our case SnowFlake which already includes a wide variety of our organizational tenants allowing us to support a wider range of use cases.
It seemed counter productive to create our team’s exclusive and robust data lake solution using something like AWS Glue/Athena to then be scrapped in favor of another tool more inline with organizational pratices.
The idea to consume it through HTTP does intrigues me, seing it as a possible low hanging fruit although again from what I’ve gathered Grafana also does not support this natively and requires again the yesoreyeram-infinity-datasource plugin. Would need to verify if in Grafana Cloud being a SaaS platform allows us to provide and set its own IAM Role, I’m aware if utilizing a native datasource (e.g. CloudWatch or Athena) there would be seemless akthough with this open source plugin would not sure TBH our bullet-proof alternative would be to expose a MS to retrieve this data and provide it through HTTP to grafana via this plugin has its datasource, although I can already foresee some waves or a bombardment of questions around data exposure even if we were to assure encryption on travel and propper authorization in place.

:person_shrugging: :person_shrugging: :person_shrugging:

So answer, which you don’t want to hear: there is no native way.

Go and request native S3 datasource: Data Source Roadmap · GitHub (I would like to see how unstructured data from S3 can be visualized :-D)

I admit… Indeed that is not the answer I would want to hear although it is pretty much the one I would expect.
No real concerns, we were simply challenging the need to create a AWS IAM User to accomudate this plugin’s AWS authentication UX (which requires long-term credential configuration) and to instead rely on a much more manageable IAM Role… Then again this being a temporary measure I suppose some form of compromise is to be expected.

Appreciate your commentary on the topic @jangaraj :slight_smile:

1 Like

You might want to check if recent Grafana versions or updates provide native support for S3 data integration via IAM roles. If not, using a dedicated plugin or the forbidden leggings considering alternative approaches for secure AWS authentication might be necessary.