Grafana docker upgrade fails

Grafana Installation
1

  • What Grafana version and what operating system are you using?
    Grafana 9.0.1 , Ubuntu 20.0.4

  • What are you trying to achieve?
    Upgrade the docker container to latest version

  • How are you trying to achieve it?
    By deleting existing container and image and loading new image using following command

echo "Container upgrade started.."
sudo docker stop grafana
echo "Container Stopped."
sudo docker rm `sudo docker ps -aqf "name=grafana"`
echo "Container Deleted."
sudo docker rmi `sudo docker images grafana/grafana -q`
echo "Container Image Deleted."


docker run -d \
        --name=grafana \
        --restart unless-stopped \
        -p 3000:3000 \
        -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-simple-json-datasource,alexanderzobnin-zabbix-app,marcusolsson-dynamictext-panel" \
        -e "GF_LOG_MODE=console file" \
        -e TZ=Asia/Kolkata \
        -v grafana-data:/var/lib/grafana \
        -v grafana-config:/etc/grafana \
        -v grafana-def-config:/usr/share/grafana/conf \
        grafana/grafana:latest

echo "Latest Container Loaded"

docker ps -aqf "name=grafana"
  • What happened?
    When the container starts after the upgrade grafana keeps restarting. Please see log below. It shows no errors but just keeps restarting
logger=settings t=2022-06-24T11:57:33.349124489+05:30 level=info msg="Starting Grafana" version=9.0.1 commit=14e988bd22 branch=HEAD compiled=2022-06-21T19:13:01+05:30
logger=settings t=2022-06-24T11:57:33.350275082+05:30 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-06-24T11:57:33.350410172+05:30 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-06-24T11:57:33.350476949+05:30 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-06-24T11:57:33.350574429+05:30 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-06-24T11:57:33.350633483+05:30 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-06-24T11:57:33.350693316+05:30 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-06-24T11:57:33.350751944+05:30 level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2022-06-24T11:57:33.350825665+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2022-06-24T11:57:33.350884942+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2022-06-24T11:57:33.350943386+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2022-06-24T11:57:33.351029273+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2022-06-24T11:57:33.351096846+05:30 level=info msg="Config overridden from Environment variable" var="GF_LOG_MODE=console file"
logger=settings t=2022-06-24T11:57:33.351154512+05:30 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-06-24T11:57:33.351214881+05:30 level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-06-24T11:57:33.351271843+05:30 level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-06-24T11:57:33.351332305+05:30 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-06-24T11:57:33.351387823+05:30 level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-06-24T11:57:33.351468192+05:30 level=info msg="App mode production"
logger=settings t=2022-06-24T11:57:58.366188735+05:30 level=info msg="Starting Grafana" version=9.0.1 commit=14e988bd22 branch=HEAD compiled=2022-06-21T19:13:01+05:30
logger=settings t=2022-06-24T11:57:58.367202589+05:30 level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2022-06-24T11:57:58.36731531+05:30 level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2022-06-24T11:57:58.367369512+05:30 level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2022-06-24T11:57:58.367431011+05:30 level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2022-06-24T11:57:58.367482899+05:30 level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2022-06-24T11:57:58.367534158+05:30 level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2022-06-24T11:57:58.367597842+05:30 level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2022-06-24T11:57:58.367651674+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2022-06-24T11:57:58.367703618+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2022-06-24T11:57:58.367754839+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2022-06-24T11:57:58.367818283+05:30 level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2022-06-24T11:57:58.367876078+05:30 level=info msg="Config overridden from Environment variable" var="GF_LOG_MODE=console file"
logger=settings t=2022-06-24T11:57:58.367939355+05:30 level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2022-06-24T11:57:58.367994021+05:30 level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2022-06-24T11:57:58.368048705+05:30 level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2022-06-24T11:57:58.368103463+05:30 level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2022-06-24T11:57:58.368161777+05:30 level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2022-06-24T11:57:58.368214425+05:30 level=info msg="App mode production"

  • What did you expect to happen?
    To strart properly

  • Can you copy/paste the configuration(s) that you are having problems with?

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No errors in the log

  • Did you follow any online instructions? If so, what is the URL?

2

hi @jinujoseph

are you still blocked on this or did the issue resolve itself?

3

@mattabrams
This is still blocked. I just cant figure out where to start debugging since there are no error messages in the logfile.

One more input: I was shocked today to find that my Internet Data for the month was completely consumed. On investigation i found that the problem was with this grafana docker instance. It was downloading almost 200GB per day. I am not sure what it was doing but this was in an infinite restart loop and every time it restarted it would download some data. Again cant seem to figure out what it is downloading. For now i have shut it down.

4

Hi Guys
Any suggestion on what is going wrong would help. How do i go about troubleshooting this since there are no error messages in the log file

5

Just to update on this. I tried the following things

  1. Fresh setup of Grafana (including new volumes) using docker works fine
  2. Fresh setup of Grafana> but using the old grafana-data volume , but new config and def-config volumes works fine
  3. In the fresh setup created in step 2 I changed the setting in grafana.ini to enable HTTPS. For this following parameters were changed in the server section - protocol, domain, cert_file and cert_key. Now if i start grafana it works fine but https is not recognized and it defaults to http. I had this problem in the old setup also. In that setup i addressed the problem by making the same changes in the defaults.ini file. But here when i make changes to the defaults.ini file the docker container goes into an infinite reboot loop.

If any body has faced this issue or know the steps for trouble shooting this.

EDIT NOTE: The issue was solved. Very stupid and basic error. I did not know that when you make changes to the grafana.ini file you have to remove the “;” at the start of the line. Doing this resolved the error.

6

I was about to post here :smiley: but glad that you found the issue as I tried your steps and could not reproduce it.

All good and have fun with grafana :grafana: