Images not rendered in Slack channel (https://slack.com/)

Hi,

I am running grafana 4.3.0 on CentOS Linux release 7.3.1611 (Core). I have configured alert notifications for slack and have configured AWS S3 for image upload. When i query the S3 bucket list, i do not see any images and the same are not rendered in the Slack Channel (https://slack.com/)

I am seeing the below error in /var/log/grafana/grafana.log

t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting Grafana” logger=main version=4.3.1 commit=befc15c compiled=2017-05-23T19:20:22+0530
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config loaded from” logger=settings file=/usr/share/grafana/conf/defaults.ini
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config loaded from” logger=settings file=/etc/grafana/grafana.ini
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.data=/var/lib/grafana"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.logs=/var/log/grafana"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Config overriden from command line” logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Home” logger=settings path=/usr/share/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Data” logger=settings path=/var/lib/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Logs” logger=settings path=/var/log/grafana
t=2017-05-25T18:50:33+0530 lvl=info msg=“Path Plugins” logger=settings path=/var/lib/grafana/plugins
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing DB” logger=sqlstore dbtype=sqlite3
t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting DB migration” logger=migrator
t=2017-05-25T18:50:33+0530 lvl=info msg=“Executing migration” logger=migrator id="copy data account to org"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Skipping migration condition not fulfilled” logger=migrator id="copy data account to org"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Executing migration” logger=migrator id="copy data account_user to org_user"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Skipping migration condition not fulfilled” logger=migrator id="copy data account_user to org_user"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Starting plugin search” logger=plugins
t=2017-05-25T18:50:33+0530 lvl=info msg=“Registering plugin” logger=plugins name="Pie Chart"
t=2017-05-25T18:50:33+0530 lvl=info msg="Initializing Stream Manager"
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing HTTP Server” logger=http.server address=0.0.0.0:3000 protocol=http subUrl= socket=
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing Alerting” logger=alerting.engine
t=2017-05-25T18:50:33+0530 lvl=info msg=“Initializing CleanUpService” logger=cleanup
t=2017-05-25T18:52:30+0530 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/plugins status=302 remote_addr=103.61.199.2 time_ms=1 size=29
t=2017-05-25T18:52:30+0530 lvl=info msg=“Request Completed” logger=context userId=0 orgId=0 uname= method=GET path=/login status=302 remote_addr=103.61.199.2 time_ms=26 size=47
t=2017-05-25T22:05:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:05:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:05:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:05:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 26EF32AC4DF560E1, host id: eIEk4ZR0zMTgKkrnDCQBS+k/zIiOKHjSe0yjbXFO8Vkn2YM1dM34zkYAWqZG++HkzK2TUnLuOys="
t=2017-05-25T22:05:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:05:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:15:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:15:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:15:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:15:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 7B891F1F05721452, host id: m6yKA9jNcmHe3ybYHBnbJevRsHIRGfezZHlxtxpID/cw4e2OLNmlT/hDV8CMjt9Uf0j8AnqWtgE="
t=2017-05-25T22:15:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:15:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:20:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:20:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:20:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:20:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: E7E6BA41ED1B4796, host id: sC75Sf9mBh7n6XlFZda5GsNOd1skHuEWsXcc/AsL2wnsCo10el2yGygE61jgW0AkT1Aguoapu0A="
t=2017-05-25T22:20:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:20:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:25:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:25:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:25:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:25:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: A80723A264FEE9FD, host id: NwwrcZVMCHQtVqLYPe9UVqzjGJfkK+g3mEftQUTSzJEVdjedybMN5T5aEMNcc9zMdiYx+KioMl0="
t=2017-05-25T22:25:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:25:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:35:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T22:35:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:35:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:35:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 5334578F520E6E6A, host id: pPPhCPNZCXBK9cudHeamMF9cJBaoXSeL+WsMPXiOqc2gv/de3rlSpvnrEF5TkC2AN2Bd/+x8e6E="
t=2017-05-25T22:35:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:35:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T22:55:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T22:55:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T22:55:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T22:55:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: F643B7DC094BF13E, host id: 8T1x7dlKzhdW/mnNK+bcRfTidcnTtSbDisnB2/i1hNBHw4oA+OqISOZ/l1Sz4jwGtJxXbd6LGIg="
t=2017-05-25T22:55:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T22:55:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:00:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T23:00:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:00:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:00:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 5FF62C65F275CCE4, host id: BzccX8Iveo4U1SGWxH4B4OBupcM3asTT+4faf2VHe+QAyccgw7rVrBcbrhA9U0i/iSsT6NIqk54="
t=2017-05-25T23:00:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:00:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:15:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=alerting prev state=ok
t=2017-05-25T23:15:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:15:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:15:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 60A4D99F391390A4, host id: PPikGg0PxT/epbqO4YroZxeRJAE9AP0abIGYXtDqsXfd+uzrt0PNE3O5Wr/EKTAcvMLbodRRN3M="
t=2017-05-25T23:15:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:15:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification="Transactional Metrics"
t=2017-05-25T23:20:01+0530 lvl=info msg=“New state change” logger=alerting.resultHandler alertId=3 newState=ok prev state=alerting
t=2017-05-25T23:20:01+0530 lvl=info msg=“Sending notifications for” logger=alerting.notifier ruleId=3 sent count=1
t=2017-05-25T23:20:01+0530 lvl=info msg=Rendering logger=png-renderer path="dashboard-solo/db/transaction-metrics?&panelId=2"
t=2017-05-25T23:20:03+0530 lvl=eror msg=“Failed to upload alert panel image.” logger=alerting.notifier error="AuthorizationHeaderMalformed: The authorization header is malformed; the region ‘us-east-1’ is wrong; expecting ‘ap-southeast-1’\n\tstatus code: 400, request id: 10BC4C55B40E282B, host id: bC1Ez7Y3ZNsCKhXRYMlCmNc2Rmvd8BzwcdszC7QVOpTVpRI7jnQylwyv9r/R4NEt1RyJnzePogU="
t=2017-05-25T23:20:03+0530 lvl=info msg=“Sending notification” logger=alerting.notifier type=slack id=2 isDefault=false
t=2017-05-25T23:20:03+0530 lvl=info msg=“Executing slack notification” logger=alerting.notifier.slack ruleId=3 notification=“Transactional Metrics”

Any help will be highly appreciable

Regards,

Kaushal

upgrade to 4.3.1, we fixed an issue with s3 uploads in that patch release

looks like your bucket urls is wrong, please update your config so that he bucket url includes the correct s3 region

Hi Torkel,

AWS S3 region is ap-southeast-1

[external_image_storage.s3]
bucket_url = http://tapzo-grafanadbteam.s3.amazonaws.com/grafana/
access_key = xxxxxxxxxxxxxxxxxxxxxxx
secret_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[root@tr3-17-18 opt]# rpm -qa | grep grafana
grafana-4.3.1-1.x86_64
[root@tr3-17-18 opt]# cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)
[root@tr3-17-18 opt]#

Please suggest further. Thanks in Advance.

Regards,

Kaushal

Yes, include region in bucket url (domain name)

Thanks Torkel and it worked like a charm

http://xxxxxxxxxx.s3-ap-southeast-1.amazonaws.com/grafana/