Troubleshoot HTTPS not working over docker container Grafana 11

Grafana Configuration
1
  • What Grafana version and what operating system are you using?
    latest grafana-oss docker image
  • What are you trying to achieve?
    configure a docker container running over HTTPS
  • How are you trying to achieve it?
docker run -d \
-p 3001:3001 --name=grafana_new \
--volume "grafana-storage:/var/lib/grafana" \
--volume "grafana-config:/etc/grafana" \
--volume "grafana-logs:/var/log/grafana" \
--volume "/home/agrimarche/grafana_ssl:/certs/" \
-e "GF_SERVER_PROTOCOL=HTTPS" \
-e "GF_SERVER_HTTP_PORT=3001" \
-e "GF_SERVER_CERT_FILE=/certs/grafana.crt" \
-e "GF_SERVER_CERT_KEY=/certs/grafana.key" \
grafana/grafana-oss
  • What happened?

Web server is responding over HTTP but not HTTPS

user@server:~$ curl https(..)//192.168.50.5:3001 --verbose
*   Trying 192.168.50.5:3001...
* Connected to 192.168.50.5 (192.168.50.5) port 3001 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* (5454) (IN), , Unknown (72):
* error:0A00010B:SSL routines::wrong version number
* Closing connection 0
curl: (35) error:0A00010B:SSL routines::wrong version number
user@server:~$ curl http://192.168.50.5:3001 --verbose
*   Trying 192.168.50.5:3001...
* Connected to 192.168.50.5 (192.168.50.5) port 3001 (#0)
> GET / HTTP/1.1
> Host: 192.168.50.5:3001
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Cache-Control: no-store
< Content-Type: text/html; charset=utf-8
< Location: /login
< X-Content-Type-Options: nosniff
< X-Frame-Options: deny
< X-Xss-Protection: 1; mode=block
< Date: Thu, 20 Jun 2024 16:50:09 GMT
< Content-Length: 29
<
<a href="/login">Found</a>.

* Connection #0 to host 192.168.50.5 left intact
  • What did you expect to happen?

Answer over HTTPS

  • Can you copy/paste the configuration(s) that you are having problems with?
    See above for docker command used to spawn the container

  • Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
    No error, but no logs regarding HTTPS neither:

logger=settings t=2024-06-20T16:35:35.27986503Z level=info msg="Starting Grafana" version=11.0.0 commit=83b9528bce85cf9371320f6d6e450916156da3f6 branch=v11.0.x compiled=2024-06-20T16:35:35Z
logger=settings t=2024-06-20T16:35:35.280888939Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini
logger=settings t=2024-06-20T16:35:35.28092714Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini
logger=settings t=2024-06-20T16:35:35.28095854Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana"
logger=settings t=2024-06-20T16:35:35.28098514Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana"
logger=settings t=2024-06-20T16:35:35.281198942Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins"
logger=settings t=2024-06-20T16:35:35.281238342Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning"
logger=settings t=2024-06-20T16:35:35.281264443Z level=info msg="Config overridden from command line" arg="default.log.mode=console"
logger=settings t=2024-06-20T16:35:35.281294043Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana"
logger=settings t=2024-06-20T16:35:35.281322043Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana"
logger=settings t=2024-06-20T16:35:35.281350743Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
logger=settings t=2024-06-20T16:35:35.281383544Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
logger=settings t=2024-06-20T16:35:35.281411144Z level=info msg="Config overridden from Environment variable" var="GF_SERVER_PROTOCOL=HTTPS"
logger=settings t=2024-06-20T16:35:35.281435044Z level=info msg="Config overridden from Environment variable" var="GF_SERVER_HTTP_PORT=3001"
logger=settings t=2024-06-20T16:35:35.281461644Z level=info msg="Config overridden from Environment variable" var="GF_SERVER_CERT_FILE=/certs/grafana.crt"
logger=settings t=2024-06-20T16:35:35.281499145Z level=info msg="Config overridden from Environment variable" var="GF_SERVER_CERT_KEY=/certs/grafana.key"
logger=settings t=2024-06-20T16:35:35.281525645Z level=info msg=Target target=[all]
logger=settings t=2024-06-20T16:35:35.281566445Z level=info msg="Path Home" path=/usr/share/grafana
logger=settings t=2024-06-20T16:35:35.281592346Z level=info msg="Path Data" path=/var/lib/grafana
logger=settings t=2024-06-20T16:35:35.281619946Z level=info msg="Path Logs" path=/var/log/grafana
logger=settings t=2024-06-20T16:35:35.281658346Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins
logger=settings t=2024-06-20T16:35:35.281695246Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning
logger=settings t=2024-06-20T16:35:35.281728647Z level=info msg="App mode production"
logger=sqlstore t=2024-06-20T16:35:35.282788656Z level=info msg="Connecting to DB" dbtype=sqlite3
logger=migrator t=2024-06-20T16:35:35.30547916Z level=info msg="Locking database"
logger=migrator t=2024-06-20T16:35:35.30552936Z level=info msg="Starting DB migrations"
logger=migrator t=2024-06-20T16:35:35.357418426Z level=info msg="migrations completed" performed=0 skipped=558 duration=1.667815ms
logger=migrator t=2024-06-20T16:35:35.358682638Z level=info msg="Unlocking database"
logger=secrets t=2024-06-20T16:35:35.360649955Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=plugin.angulardetectorsprovider.dynamic t=2024-06-20T16:35:35.534657918Z level=info msg="Restored cache from database" duration=944.409µs
logger=plugin.store t=2024-06-20T16:35:35.537694845Z level=info msg="Loading plugins..."
logger=plugins.registration t=2024-06-20T16:35:35.689493308Z level=error msg="Could not register plugin" pluginId=xychart error="plugin xychart is already registered"
logger=plugins.initialization t=2024-06-20T16:35:35.689547108Z level=error msg="Could not initialize plugin" pluginId=xychart error="plugin xychart is already registered"
logger=local.finder t=2024-06-20T16:35:35.68978521Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled
logger=plugin.store t=2024-06-20T16:35:35.689846211Z level=info msg="Plugins loaded" count=54 duration=152.151466ms
logger=query_data t=2024-06-20T16:35:35.700473406Z level=info msg="Query Service initialization"
logger=live.push_http t=2024-06-20T16:35:35.706867364Z level=info msg="Live Push Gateway initialization"
logger=ngalert.notifier.alertmanager org=1 t=2024-06-20T16:35:35.714791135Z level=info msg="Applying new configuration to Alertmanager" configHash=a013a3f424edb13bed8050eaf374d506
logger=ngalert.state.manager t=2024-06-20T16:35:35.733924307Z level=info msg="Running in alternative execution of Error/NoData mode"
logger=infra.usagestats.collector t=2024-06-20T16:35:35.738674249Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.alerting t=2024-06-20T16:35:35.739233954Z level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2024-06-20T16:35:35.739277255Z level=info msg="finished to provision alerting"
logger=grafanaStorageLogger t=2024-06-20T16:35:35.739780359Z level=info msg="Storage starting"
logger=provisioning.dashboard t=2024-06-20T16:35:35.741775777Z level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2024-06-20T16:35:35.741819177Z level=info msg="finished to provision dashboards"
logger=ngalert.state.manager t=2024-06-20T16:35:35.745550011Z level=info msg="Warming state cache for startup"
logger=ngalert.state.manager t=2024-06-20T16:35:35.746454219Z level=info msg="State cache has been initialized" states=0 duration=898.908µs
logger=ngalert.multiorg.alertmanager t=2024-06-20T16:35:35.747077225Z level=info msg="Starting MultiOrg Alertmanager"
logger=ngalert.scheduler t=2024-06-20T16:35:35.747441628Z level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1
logger=ticker t=2024-06-20T16:35:35.747842031Z level=info msg=starting first_tick=2024-06-20T16:35:40Z
logger=http.server t=2024-06-20T16:35:35.751969069Z level=info msg="HTTP Server Listen" address=[::]:3001 protocol=http subUrl= socket=
logger=grafana.update.checker t=2024-06-20T16:35:35.911156598Z level=info msg="Update check succeeded" duration=169.25522ms
logger=plugins.update.checker t=2024-06-20T16:35:35.920398781Z level=info msg="Update check succeeded" duration=176.642586ms
logger=grafana-apiserver t=2024-06-20T16:35:37.583018307Z level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-06-20T16:35:37.584315819Z level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
logger=infra.usagestats t=2024-06-20T16:37:18.754520267Z level=info msg="Usage stats are ready to report"
logger=context userId=0 orgId=0 uname= t=2024-06-20T16:42:08.96830551Z level=info msg="Request Completed" method=GET path=/ status=302 remote_addr=192.168.50.5 time_ms=0 duration=154.401µs size=29 referer= handler=/ status_source=server
logger=cleanup t=2024-06-20T16:45:35.842101729Z level=info msg="Completed cleanup jobs" duration=94.068745ms
logger=plugins.update.checker t=2024-06-20T16:45:36.220629029Z level=info msg="Update check succeeded" duration=295.231452ms

screenshot of my grafana container parameters: Imgur: The magic of the Internet

  • Did you follow any online instructions? If so, what is the URL?
    mix of docs/grafana/latest/setup-grafana/installation/docker/
    and docs/grafana/latest/setup-grafana/configure-grafana/
1 Like
2

So it really didn’t start https.

Question is why it didn’t apply env variables.

I would try for debugging:

  • use config file instead of env variables (you are mixing both atm)
  • older 10.x version
  • increase log level to debug
  • make cert/key readable for everyone
    …
3

Actually, it’s case sensitive, so you need:

-e "GF_SERVER_PROTOCOL=https" \
2 Likes
4

You really are a champion @jangaraj

Thank you all for your time

2 Likes