How do I setup IAM Authentication for a backing RDS mysql database?

Grafana MySQL
1

I have an ELK stack with grafana installed on EC2 instances. I’ve created a mysql database in RDS, created a grafana database and a grafana user with the appropriate permissions. That user is able to be accessed via mysql utilizing an appropriately setup IAM role.

However, when I try to start the Grafana service using that database I get the following error in the logs:

“Fail to initialize orm engine” logger=sqlstore error=“Sqlstore::Migration failed err: Error 1045: Access denied for user ‘grafana’@‘SERVERIP’ (using password: YES)\n”

My grafana.ini database section looks like:

[database]
host = aurora-cluster-ClusterName.cluster-RandomAWSSuppliedChars.Region.rds.amazonaws.com:3306
type = mysql
name = grafanaDbName
user = grafanaUserName
password = aws rds generate-db-auth-token --hostname aurora-cluster-ClusterName.cluster-RandomAWSSuppliedChars.Region.rds.amazonaws.com --username grafanaUserName --port 3306 --region Region

I’ve tried supplying no password as well, but I’m pretty sure I need the generated auth token to be supplied. I also tried generating an auth token and pasting it in manually during the 15 minute window while it’s good for testing purposes and I got the same Access denied error.

Is IAM Authentication supported? If so, how do I supply the required Auth token?

2

Hi,

No execution of binary will happen in .ini file so the following will not execute:

aws rds generate-db-auth-token --hostname aurora-cluster-ClusterName.cluster-RandomAWSSuppliedChars.Region.rds.amazonaws.com --username grafanaUserName --port 3306 --region Region

Suggest you to use environment variable for that configuration.

Reading aws documentation seems like you need to provide the SSL certificate file that contains the public key.

See Grafana configuration documentation regarding ssl.

Marcus

3

Thanks, that was the information I needed to get a db connection.

Now I’m seeing the db migration fail to start, but that’s no longer a connection issue.

4

Sounds like a permission problem. Please change the log level to debug, see documentation and check grafana server log. What’s the db migration error you got?

5

I ended up giving up on IAM auth. I changed the authentication to use a standard un/pw pair and the connection worked as expected.

While it did appear that the IAM auth was successful the grafana service would get to the step where it tried to begin the db migration and then crash with no further information. I tried setting the log level to debug, but I didn’t see any actual debug statements in the grafana.log file.

While I’m glad that the traditional method of authentication works, it’s a bit frustrating that the process to implement the more secure IAM alternative is significantly more opaque.

6

Hi I’m also trying to get up AWS IAM with an Aurora RDS as Grafana database in grafana.ini.

I’ve set up the following for the database section. The password “AwsSuppliedToken” is supplied by the environmental variable using “export GF_DATABASE_PASSWORD=”$(aws rds generate-db-auth-token --hostname $RDSHOST --port 3306 --region us-east-1 --username jdoe)"

    [database]
    type = mysql
    host = my_aurora.rds.amazonaws.com:3306
    name = GrafanaDB
    user = jdoe
    password = AwsSuppliedToken
    ssl_mode=true
    ca_cert_path =/var/log/grafana/rds-combined-ca-bundle.pem
    server_cert_name=my_aurora.rds.amazonaws.com

I got the following error message:

lvl=eror msg=“Fail to initialize orm engine” logger=sqlstore error=“Sqlstore::Migration failed err: this user requires clear text authentication. If you still want to use it, please add ‘allowCleartextPasswords=1’ to your DSN\n”

First, I thought the “allowCleartextPasswords=true” is by default (https://github.com/grafana/grafana/blob/master/vendor/github.com/go-sql-driver/mysql/dsn.go)

Second, I don’t think using the [session] section is possible because my password (ie. the AWS IAM token) is a super long string with random characters (hence I had to use the environmental variable in the previous step). Essentially I can’t use:
jdoe:password@tcp(127.0.0.1:3306)/GrafanaDB?allowCleartextPasswords=true

3rd, I tried to hack the grafana.ini file by including dsn options in the following manner:

            [database]
            type = mysql
            host = my_aurora.rds.amazonaws.com:3306
            name = GrafanaDB?allowCleartextPasswords=true

But it seems there’s hard-coded dsn option “?collation=utf8mb4_unicode_ci” already, so my hack doesn’t work:

lvl=eror msg=“Fail to initialize orm engine” logger=sqlstore error=“Sqlstore::Migration failed err: invalid bool value: true?collation=utf8mb4_unicode_ci\n”

I’d like to know if there’s another way to pass in the dsn options, specifically the ‘allowCleartextPasswords’ option. That’s probably the last thing required to gain access to AWS RDS using IAM in grafana.ini.

Thanks!