Snapshot image creation fails. PhantomJS QNetworkReply::NetworkError

khallela · August 27, 2019, 6:17am

I am running Grafana v6.3.3 (commit: 84d0a71) via the official grafana/grafana:6.3.3 Docker image. It includes PhantomJS version 2.1.1.

I tried this with grafana/grafana:6.2.0 as well, and had similar results.

I have Grafana configured with an S3 bucket as the external snapshot storage, but I have tried this with local as well.

When I visit a dashboard and try to generate a snapshot, I get the “Grafana / Server Error” page with a message that says “Rendering failed.”

I see something like this in the Grafana logs:

t=2019-08-28T04:29:00+0000 lvl=info msg=Rendering logger=rendering path="d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2&orgId=1&from=1566944939228&to=1566966539228&width=1000&height=500&tz=Asia%2FSaigon"
t=2019-08-28T04:29:00+0000 lvl=dbug msg="executing Phantomjs" logger=rendering binPath=/usr/share/grafana/tools/phantomjs/phantomjs cmdArgs="[--ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=false /usr/share/grafana/tools/phantomjs/render.js url=http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2&orgId=1&from=1566944939228&to=1566966539228&width=1000&height=500&tz=Asia%2FSaigon&render=1 width=1000 height=500 png=/var/lib/grafana/png/4LFDe2nowB9C7AlySBDK.png domain=localhost timeout=60 renderKey=bHcGj23RQVzMm1ubjT4Or7UcuJScatiT]" timezone=Asia/Saigon
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Scheduler: Putting job on to exec queue" logger=alerting.scheduler name="File descriptors used is abnormally high" id=2
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Creating new client" logger=tsdb.elasticsearch.client version=70 timeField=@timestamp indices=metricbeat-*
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Executing multisearch" logger=tsdb.elasticsearch.client search requests=1
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Encoding batch requests to json" logger=tsdb.elasticsearch.client batch requests=1
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Encoded batch requests to json" logger=tsdb.elasticsearch.client took=144.294µs
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Executing request" logger=tsdb.elasticsearch.client url="https://elasticsearch.XXX.XXX.com:9200/_msearch?max_concurrent_shard_requests=5" method=POST
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Request configured to use basic authentication" logger=tsdb.elasticsearch.client
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Executed request" logger=tsdb.elasticsearch.client took=84.359632ms
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Received multisearch response" logger=tsdb.elasticsearch.client code=200 status="200 OK" content-length=393007
t=2019-08-28T04:29:01+0000 lvl=dbug msg="Decoding multisearch json response" logger=tsdb.elasticsearch.client
t=2019-08-28T04:29:02+0000 lvl=dbug msg="Decoded multisearch json response" logger=tsdb.elasticsearch.client took=214.212203ms
t=2019-08-28T04:29:02+0000 lvl=dbug msg="Job Execution completed" logger=alerting.engine timeMs=-598.462 alertId=2 name="File descriptors used is abnormally high" firing=false attemptID=1
t=2019-08-28T04:29:03+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=1
t=2019-08-28T04:29:08+0000 lvl=dbug msg="Phantomjs output" logger=rendering out=
t=2019-08-28T04:29:08+0000 lvl=dbug msg="Phantomjs error" logger=rendering error="signal: killed"
t=2019-08-28T04:29:08+0000 lvl=eror msg="Phantomjs exited with non zero exit code" logger=rendering error="signal: killed"
t=2019-08-28T04:29:08+0000 lvl=eror msg="Rendering failed." logger=context userId=1 orgId=1 uname=admin error="signal: killed"
t=2019-08-28T04:29:08+0000 lvl=eror msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=GET path=/render/d-solo/wleFcFKZz/rabbitmq-system-alerts status=500 remote_addr=10.90.0.6 time_ms=7318 size=1722 referer="http://grafana.XXX.com/d/wleFcFKZz/rabbitmq-system-alerts?panelId=2&fullscreen&edit&orgId=1"

When I enter the Grafana container and execute the PhantomJS command shown above with debug=true, I get something like this:

grafana@grafana-9757c88ff-fjsjd:/usr/share/grafana$ /usr/share/grafana/tools/phantomjs/phantomjs --ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=true /usr/share/grafana/tools/phantomjs/render.js url=http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2&orgId=1&from=1566944939228&to=1566966539228&width=1000&height=500&tz=Asia%2FSaigon&render=1 width=1000 height=500 png=/var/lib/grafana/png/4LFDe2nowB9C7AlySBDK.png domain=localhost timeout=60 renderKey=bHcGj23RQVzMm1ubjT4Or7UcuJScatiT
[1] 187
[2] 188
[3] 189
[4] 190
[5] 191
[6] 192
[7] 193
[3]   Done                    from=1566944939228
[4]   Done                    to=1566966539228
grafana@grafana-9757c88ff-fjsjd:/usr/share/grafana$ 2019-08-28T04:33:54 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persistent cookie storage)
2019-08-28T04:33:56 [DEBUG] Set  "http"  proxy to:  "" : 1080
2019-08-28T04:33:56 [DEBUG] Phantom - execute: Configuration
2019-08-28T04:33:56 [DEBUG]      0 objectName : ""
2019-08-28T04:33:56 [DEBUG]      1 cookiesFile : ""
2019-08-28T04:33:56 [DEBUG]      2 diskCacheEnabled : "false"
2019-08-28T04:33:56 [DEBUG]      3 maxDiskCacheSize : "-1"
2019-08-28T04:33:56 [DEBUG]      4 diskCachePath : ""
2019-08-28T04:33:56 [DEBUG]      5 ignoreSslErrors : "true"
2019-08-28T04:33:56 [DEBUG]      6 localUrlAccessEnabled : "false"
2019-08-28T04:33:56 [DEBUG]      7 localToRemoteUrlAccessEnabled : "false"
2019-08-28T04:33:56 [DEBUG]      8 outputEncoding : "UTF-8"
2019-08-28T04:33:56 [DEBUG]      9 proxyType : "http"
2019-08-28T04:33:56 [DEBUG]      10 proxy : ":1080"
2019-08-28T04:33:56 [DEBUG]      11 proxyAuth : ":"
2019-08-28T04:33:56 [DEBUG]      12 scriptEncoding : "UTF-8"
2019-08-28T04:33:56 [DEBUG]      13 webSecurityEnabled : "true"
2019-08-28T04:33:56 [DEBUG]      14 offlineStoragePath : ""
2019-08-28T04:33:56 [DEBUG]      15 localStoragePath : ""
2019-08-28T04:33:56 [DEBUG]      16 localStorageDefaultQuota : "-1"
2019-08-28T04:33:56 [DEBUG]      17 offlineStorageDefaultQuota : "-1"
2019-08-28T04:33:56 [DEBUG]      18 printDebugMessages : "true"
2019-08-28T04:33:56 [DEBUG]      19 javascriptCanOpenWindows : "true"
2019-08-28T04:33:56 [DEBUG]      20 javascriptCanCloseWindows : "true"
2019-08-28T04:33:56 [DEBUG]      21 sslProtocol : "default"
2019-08-28T04:33:56 [DEBUG]      22 sslCiphers : "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:RC4-SHA:RC4-MD5"
2019-08-28T04:33:56 [DEBUG]      23 sslCertificatesPath : ""
2019-08-28T04:33:56 [DEBUG]      24 sslClientCertificateFile : ""
2019-08-28T04:33:56 [DEBUG]      25 sslClientKeyFile : ""
2019-08-28T04:33:56 [DEBUG]      26 sslClientKeyPassphrase : ""
2019-08-28T04:33:56 [DEBUG]      27 webdriver : ":"
2019-08-28T04:33:56 [DEBUG]      28 webdriverLogFile : ""
2019-08-28T04:33:56 [DEBUG]      29 webdriverLogLevel : "INFO"
2019-08-28T04:33:56 [DEBUG]      30 webdriverSeleniumGridHub : ""
2019-08-28T04:33:56 [DEBUG] Phantom - execute: Script & Arguments
2019-08-28T04:33:56 [DEBUG]      script: "/usr/share/grafana/tools/phantomjs/render.js"
2019-08-28T04:33:56 [DEBUG]      0 arg: "url=http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2"
2019-08-28T04:33:56 [DEBUG] Phantom - execute: Starting normal mode
2019-08-28T04:33:56 [DEBUG] WebPage - setupFrame ""
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r")))
url=<url> png=<filename> width=<width> height=<height> renderKey=<key>
2019-08-28T04:33:56 [DEBUG] WebPage - updateLoadingProgress: 10
2019-08-28T04:33:56 [DEBUG] WebPage - updateLoadingProgress: 100
2019-08-28T04:33:56 [DEBUG] WebPage - setupFrame ""
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] WebPage - updateLoadingProgress: 10
2019-08-28T04:33:56 [DEBUG] WebPage - setupFrame ""
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T04:33:56 [DEBUG] WebPage - updateLoadingProgress: 100
2019-08-28T04:33:56 [DEBUG] WebPage - updateLoadingProgress: 10
2019-08-28T04:33:56 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(OperationCanceledError) ( "Operation canceled" ) URL: "http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2"

[1]   Done                    /usr/share/grafana/tools/phantomjs/phantomjs --ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=true /usr/share/grafana/tools/phantomjs/render.js url=http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2
[2]   Done                    orgId=1
[5]   Done                    width=1000
[6]-  Done                    height=500
[7]+  Done                    tz=Asia%2FSaigon

If I curl that URL shown above from within the container, I get a 302 Found:

grafana@grafana-9757c88ff-fjsjd:/usr/share/grafana$ curl -I http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?panelId=2
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Location: /login
Pragma: no-cache
Set-Cookie: redirect_to=%252Fd-solo%252FwleFcFKZz%252Frabbitmq-system-alerts%253FpanelId%253D2; Path=/; HttpOnly
X-Frame-Options: deny
Date: Wed, 28 Aug 2019 04:34:51 GMT

khallela · August 28, 2019, 2:30pm

Again, this time with single quotes around the url parameter:

grafana@grafana-9757c88ff-fjsjd:/usr/share/grafana$ /usr/share/grafana/tools/phantomjs/phantomjs --ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=true /usr/share/grafana/tools/phantomjs/render.js url='http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?orgId=1&from=1566940886821&to=1566962486821&panelId=2&width=1000&height=500&tz=Asia%2FSaigon&render=1' width=1000 height=500 png=/var/lib/grafana/png/Jbu9SQjBtY3MXO232ajb.png domain=localhost timeout=60 renderKey=hO2D5vw2REnPHdmeGOLBAzGHA3YrNKC6
2019-08-28T13:46:04 [DEBUG] CookieJar - Created but will not store cookies (use option '--cookies-file=<filename>' to enable persistent cookie storage)

2019-08-28T13:46:06 [DEBUG] Set  "http"  proxy to:  "" : 1080
2019-08-28T13:46:06 [DEBUG] Phantom - execute: Configuration
2019-08-28T13:46:06 [DEBUG]      0 objectName : ""
2019-08-28T13:46:06 [DEBUG]      1 cookiesFile : ""
2019-08-28T13:46:06 [DEBUG]      2 diskCacheEnabled : "false"
2019-08-28T13:46:06 [DEBUG]      3 maxDiskCacheSize : "-1"
2019-08-28T13:46:06 [DEBUG]      4 diskCachePath : ""
2019-08-28T13:46:06 [DEBUG]      5 ignoreSslErrors : "true"
2019-08-28T13:46:06 [DEBUG]      6 localUrlAccessEnabled : "false"
2019-08-28T13:46:06 [DEBUG]      7 localToRemoteUrlAccessEnabled : "false"
2019-08-28T13:46:06 [DEBUG]      8 outputEncoding : "UTF-8"
2019-08-28T13:46:06 [DEBUG]      9 proxyType : "http"
2019-08-28T13:46:06 [DEBUG]      10 proxy : ":1080"
2019-08-28T13:46:06 [DEBUG]      11 proxyAuth : ":"
2019-08-28T13:46:06 [DEBUG]      12 scriptEncoding : "UTF-8"
2019-08-28T13:46:06 [DEBUG]      13 webSecurityEnabled : "true"
2019-08-28T13:46:06 [DEBUG]      14 offlineStoragePath : ""
2019-08-28T13:46:06 [DEBUG]      15 localStoragePath : ""
2019-08-28T13:46:06 [DEBUG]      16 localStorageDefaultQuota : "-1"
2019-08-28T13:46:06 [DEBUG]      17 offlineStorageDefaultQuota : "-1"
2019-08-28T13:46:06 [DEBUG]      18 printDebugMessages : "true"
2019-08-28T13:46:06 [DEBUG]      19 javascriptCanOpenWindows : "true"
2019-08-28T13:46:06 [DEBUG]      20 javascriptCanCloseWindows : "true"
2019-08-28T13:46:06 [DEBUG]      21 sslProtocol : "default"
2019-08-28T13:46:06 [DEBUG]      22 sslCiphers : "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-RC4-SHA:ECDHE-RSA-RC4-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:RC4-SHA:RC4-MD5"
2019-08-28T13:46:06 [DEBUG]      23 sslCertificatesPath : ""
2019-08-28T13:46:06 [DEBUG]      24 sslClientCertificateFile : ""
2019-08-28T13:46:06 [DEBUG]      25 sslClientKeyFile : ""
2019-08-28T13:46:06 [DEBUG]      26 sslClientKeyPassphrase : ""
2019-08-28T13:46:06 [DEBUG]      27 webdriver : ":"
2019-08-28T13:46:06 [DEBUG]      28 webdriverLogFile : ""
2019-08-28T13:46:06 [DEBUG]      29 webdriverLogLevel : "INFO"
2019-08-28T13:46:06 [DEBUG]      30 webdriverSeleniumGridHub : ""
2019-08-28T13:46:06 [DEBUG] Phantom - execute: Script & Arguments
2019-08-28T13:46:06 [DEBUG]      script: "/usr/share/grafana/tools/phantomjs/render.js"
2019-08-28T13:46:06 [DEBUG]      0 arg: "url=http://grafana.staged-by-discourse.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?orgId=1&from=1566940886821&to=1566962486821&panelId=2&width=1000&height=500&tz=Asia%2FSaigon&render=1"
2019-08-28T13:46:06 [DEBUG]      1 arg: "width=1000"
2019-08-28T13:46:06 [DEBUG]      2 arg: "height=500"
2019-08-28T13:46:06 [DEBUG]      3 arg: "png=/var/lib/grafana/png/Jbu9SQjBtY3MXO232ajb.png"
2019-08-28T13:46:06 [DEBUG]      4 arg: "domain=localhost"
2019-08-28T13:46:06 [DEBUG]      5 arg: "timeout=60"
2019-08-28T13:46:06 [DEBUG]      6 arg: "renderKey=hO2D5vw2REnPHdmeGOLBAzGHA3YrNKC6"
2019-08-28T13:46:06 [DEBUG] Phantom - execute: Starting normal mode
2019-08-28T13:46:06 [DEBUG] WebPage - setupFrame ""
2019-08-28T13:46:06 [DEBUG] FileSystem - _open: ":/modules/fs.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T13:46:06 [DEBUG] FileSystem - _open: ":/modules/system.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T13:46:06 [DEBUG] FileSystem - _open: ":/modules/webpage.js" QMap(("mode", QVariant(QString, "r")))
2019-08-28T13:46:06 [DEBUG] CookieJar - Saved "renderKey=hO2D5vw2REnPHdmeGOLBAzGHA3YrNKC6; domain=.localhost; path=/"
2019-08-28T13:46:06 [DEBUG] CookieJar - Rejected Cookie "renderKey=hO2D5vw2REnPHdmeGOLBAzGHA3YrNKC6; domain=localhost"
2019-08-28T13:46:06 [DEBUG] WebPage - updateLoadingProgress: 10
2019-08-28T13:46:06 [DEBUG] WebPage - updateLoadingProgress: 50
2019-08-28T13:46:06 [DEBUG] Network - Resource request error: QNetworkReply::NetworkError(AuthenticationRequiredError) ( "Host requires authentication" ) URL: "http://elastic:changeme@community.grafana.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?orgId=1&from=1566940886821&to=1566962486821&panelId=2&width=1000&height=500&tz=Asia%2FSaigon&render=1"
2019-08-28T13:46:07 [DEBUG] WebPage - updateLoadingProgress: 100
Loading a web page: http://elastic:changeme@community.grafana.com/d-solo/wleFcFKZz/rabbitmq-system-alerts?orgId=1&from=1566940886821&to=1566962486821&panelId=2&width=1000&height=500&tz=Asia%2FSaigon&render=1 status: success 60000
2019-08-28T13:46:07 [DEBUG] WebPage - setupFrame ""
2019-08-28T13:46:07 [DEBUG] WebPage - setupFrame ""
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, false)
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, false)
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, false)
2019-08-28T13:46:07 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"

[...]

2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"
2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, false)
2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n          var panelCount = document.querySelectorAll('plugin-component').length;\n          return window.panelsRendered >= panelCount;\n        })(); })()"
2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript result QVariant(bool, false)
2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript "(function() { return (function () {\n            var container = document.getElementsByClassName(\"dashboard-container\")\n            if (container.length == 0) {\n               container = document.getElementsByClassName(\"panel-container\")\n            }\n            return container[0].getBoundingClientRect();\n          })(); })()"
ERROR: TypeError: undefined is not an object (evaluating 'container[0].getBoundingClientRect')
TRACE:
 -> undefined: 6
 -> : 7

  phantomjs://code/render.js:48 in onError
2019-08-28T13:47:08 [DEBUG] WebPage - evaluateJavaScript result QVariant(Invalid)
TypeError: null is not an object (evaluating 'bb.width')

  phantomjs://code/render.js:68 in checkIsReady

khallela · August 29, 2019, 3:21am

Sometimes Grafana shuts down completely:

t=2019-08-29T03:19:31+0000 lvl=info msg=Rendering logger=rendering path="d-solo/IE4ksZFZz/rabbitmq-queue-alerts?orgId=1&from=1567037968728&to=1567048768728&panelId=2&width=1000&height=500&tz=Asia%2FSaigon"
t=2019-08-29T03:19:31+0000 lvl=dbug msg="executing Phantomjs" logger=rendering binPath=/usr/share/grafana/tools/phantomjs/phantomjs cmdArgs="[--ignore-ssl-errors=true --web-security=true --local-url-access=false --debug=false /usr/share/grafana/tools/phantomjs/render.js url=http://grafana.staged-by-discourse.com/d-solo/IE4ksZFZz/rabbitmq-queue-alerts?orgId=1&from=1567037968728&to=1567048768728&panelId=2&width=1000&height=500&tz=Asia%2FSaigon&render=1 width=1000 height=500 png=/var/lib/grafana/png/Ltlg1YW3dxXrAv0qSb3a.png domain=localhost timeout=60 renderKey=iiy3BCpNokWYhv9Q1mVLHq4a0mWeq4pe]" timezone=Asia/Saigon
t=2019-08-29T03:19:34+0000 lvl=dbug msg="Scheduling update" logger=alerting.scheduler ruleCount=1
t=2019-08-29T03:19:40+0000 lvl=info msg="Shutdown started" logger=server reason="System signal: terminated"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped UsageStatsService" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped PluginManager" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped UserAuthTokenService" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped Stream Manager"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped NotificationService" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped CleanUpService" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped TracingService" logger=server reason=nil
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped RenderingService" logger=server reason=nil
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped RemoteCache" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped InternalMetricsService" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=dbug msg="Stop polling for dashboard changes" logger=provisioning
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped provisioningServiceImpl" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped AlertEngine" logger=server reason="context canceled"
t=2019-08-29T03:19:40+0000 lvl=dbug msg="server was shutdown gracefully" logger=http.server
t=2019-08-29T03:19:40+0000 lvl=info msg="Stopped HTTPServer" logger=server reason=nil
t=2019-08-29T03:19:40+0000 lvl=eror msg="Server shutdown" logger=server reason="System signal: terminated"

khallela · August 29, 2019, 5:33am

The issue was that the container was running out of memory. I increased the memory limit beyond 100MB and the snapshots are now working.