-
What Grafana version and what operating system are you using?
Latest Windows MSI Installer, downloaded yesterday.
Windows Server 2016, not my choice.
-
What are you trying to achieve?
I am not able to use NSSM to create the service in Windows. This is due to security requirements for our air-gapped network.
I can create the Windows Service via PowerShell:
# Create the new Windows Service, with a prompt to confirm.
New-Service -Name "Grafana" -BinaryPathName "D:\GrafanaLabs\grafana\bin\grafana-server.exe" -DisplayName "Grafana" -Description "Grafana Server" -StartupType Automatic -Confirm
# Creating the service does not start it, let's do that now.
Start-Service -Name "Grafana"
# Verify it is up and running.
Get-Service -Name "Grafana"
This gives me the following registry entries:
Get-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Services\Grafana\
Type : 16
Start : 2
ErrorControl : 1
ImagePath : D:\GrafanaLabs\grafana\bin\grafana-server.exe
DisplayName : Grafana
ObjectName : LocalSystem
Description : Grafana Server
PSPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Grafana\
PSParentPath : Microsoft.PowerShell.Core\Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services
PSChildName : Grafana
PSDrive : HKLM
PSProvider : Microsoft.PowerShell.Core\Registry
-
How are you trying to achieve it?
Using PowerShell 5.1 for maximum compatibility.
-
What happened?
I can start Grafana-Server.exe manually, no problems there.
But starting it as a Windows Service fails.
Start-Service -Name "Grafana"
Start-Service : Service 'Grafana (Grafana)' cannot be started due to the following error: Cannot start service Grafana
on computer '.'.
At line:1 char:1
+ Start-Service -Name "Grafana"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OpenError: (System.ServiceProcess.ServiceController:ServiceController) [Start-Service],
ServiceCommandException
+ FullyQualifiedErrorId : CouldNotStartService,Microsoft.PowerShell.Commands.StartServiceCommand
Windows Event Log → System shows:
EventID 7009:
A timeout was reached (30000 milliseconds) while waiting for the Grafana service to connect.
And within the same second, EventID 7000:
The Grafana service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.
None of this really gives more info as to what happens.
The Grafana log in D:\GrafanaLabs\grafana\data\log does not get updated, which leads me to believe the service start doesn’t even make it as far as attempting to start the service.
To the best of my knowledge, there isn’t anything blocking the unsigned executable from running as a service. I can start the executable manually.
-
What did you expect to happen?
The Grafana service to start as expected using the local system service.
-
Can you copy/paste the configuration(s) that you are having problems with?
Currently using the default.ini from the installation, copied it to make custom.ini.
-
Did you receive any errors in the Grafana UI or in related logs? If so, please tell us exactly what they were.
Literally no errors or lines in the log.
-
Did you follow any online instructions? If so, what is the URL?
I have not found any specific to a clean install. Auth.jwt is not enabled by default.
[auth.jwt]
enabled = false
if you have it installed as MSI, it is probably already running as a service. You might want to go the route of downloading the zip and do your thing with PS
When I switched the new service to use my own domain account with local admin powers, I got some activity in the grafana.log.
logger=settings t=2024-03-27T11:07:15.2245197-05:00 level=info msg="Starting Grafana" version=10.4.1 commit=d3ce857c0eb86f571ffa993a9cd8493b6f47b630 branch=HEAD compiled=2024-03-27T11:07:15-05:00
logger=settings t=2024-03-27T11:07:15.2254898-05:00 level=info msg="Config loaded from" file=D:\GrafanaLabs\grafana\/conf/defaults.ini
logger=settings t=2024-03-27T11:07:15.2254898-05:00 level=info msg="Config loaded from" file=D:\GrafanaLabs\grafana\conf\custom.ini
logger=settings t=2024-03-27T11:07:15.2254898-05:00 level=info msg=Target target=[all]
logger=settings t=2024-03-27T11:07:15.2254898-05:00 level=info msg="Path Home" path=D:\GrafanaLabs\grafana\
logger=settings t=2024-03-27T11:07:15.2264893-05:00 level=info msg="Path Data" path=D:\GrafanaLabs\grafana\data
logger=settings t=2024-03-27T11:07:15.2264893-05:00 level=info msg="Path Logs" path=D:\GrafanaLabs\grafana\data\log
logger=settings t=2024-03-27T11:07:15.2264893-05:00 level=info msg="Path Plugins" path=D:\GrafanaLabs\grafana\data\plugins
logger=settings t=2024-03-27T11:07:15.2274891-05:00 level=info msg="Path Provisioning" path=D:\GrafanaLabs\grafana\conf\provisioning
logger=settings t=2024-03-27T11:07:15.2274891-05:00 level=info msg="App mode production"
logger=sqlstore t=2024-03-27T11:07:15.2274891-05:00 level=info msg="Connecting to DB" dbtype=sqlite3
logger=sqlstore t=2024-03-27T11:07:15.2274891-05:00 level=warn msg="SQLite database file has broader permissions than it should" path=D:\GrafanaLabs\grafana\data\grafana.db mode=-rw-rw-rw- expected=-rw-r-----
logger=migrator t=2024-03-27T11:07:15.2284893-05:00 level=info msg="Starting DB migrations"
logger=migrator t=2024-03-27T11:07:15.2394901-05:00 level=info msg="migrations completed" performed=0 skipped=548 duration=999.8µs
logger=secrets t=2024-03-27T11:07:15.2404903-05:00 level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1
logger=renderer.manager t=2024-03-27T11:07:15.2554915-05:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=plugin.store t=2024-03-27T11:07:15.2584917-05:00 level=info msg="Loading plugins..."
logger=plugin.sources t=2024-03-27T11:07:15.2594919-05:00 level=error msg="Failed to load external plugins" error="failed to open plugins path"
logger=plugins.registration t=2024-03-27T11:07:15.5425201-05:00 level=info msg="Plugin registered" pluginId=input
logger=plugin.store t=2024-03-27T11:07:15.5425201-05:00 level=info msg="Plugins loaded" count=56 duration=284.0189ms
logger=query_data t=2024-03-27T11:07:15.5505211-05:00 level=info msg="Query Service initialization"
logger=live.push_http t=2024-03-27T11:07:15.5545231-05:00 level=info msg="Live Push Gateway initialization"
logger=ngalert.migration t=2024-03-27T11:07:15.5655231-05:00 level=info msg=Starting
logger=ngalert.state.manager t=2024-03-27T11:07:15.5978489-05:00 level=info msg="Running in alternative execution of Error/NoData mode"
logger=infra.usagestats.collector t=2024-03-27T11:07:15.6008519-05:00 level=info msg="registering usage stat providers" usageStatsProvidersLen=2
logger=provisioning.alerting t=2024-03-27T11:07:15.6038495-05:00 level=info msg="starting to provision alerting"
logger=provisioning.alerting t=2024-03-27T11:07:15.6038495-05:00 level=info msg="finished to provision alerting"
logger=renderer.manager t=2024-03-27T11:07:15.6050626-05:00 level=error msg="Failed to get renderer plugin sources" error="failed to open plugins path"
logger=grafanaStorageLogger t=2024-03-27T11:07:15.6058491-05:00 level=info msg="Storage starting"
logger=ngalert.state.manager t=2024-03-27T11:07:15.6058491-05:00 level=info msg="Warming state cache for startup"
logger=ngalert.state.manager t=2024-03-27T11:07:15.6088494-05:00 level=info msg="State cache has been initialized" states=0 duration=3.0001ms
logger=secret.migration t=2024-03-27T11:07:15.6058491-05:00 level=error msg="Server lock for secret migration already exists"
logger=ngalert.multiorg.alertmanager t=2024-03-27T11:07:15.6068494-05:00 level=info msg="Starting MultiOrg Alertmanager"
logger=http.server t=2024-03-27T11:07:15.6088494-05:00 level=info msg="HTTP Server Listen" address=[::]:3000 protocol=http subUrl= socket=
logger=provisioning.dashboard t=2024-03-27T11:07:15.6068494-05:00 level=info msg="starting to provision dashboards"
logger=provisioning.dashboard t=2024-03-27T11:07:15.6128468-05:00 level=info msg="finished to provision dashboards"
logger=ngalert.scheduler t=2024-03-27T11:07:15.6098473-05:00 level=info msg="Starting scheduler" tickInterval=10s maxAttempts=1
logger=ticker t=2024-03-27T11:07:15.6138478-05:00 level=info msg=starting first_tick=2024-03-27T11:07:20-05:00
logger=plugin.signature.key_retriever t=2024-03-27T11:07:15.6218484-05:00 level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": read tcp 172.18.152.26:49414->34.120.177.193:443: wsarecv: An existing connection was forcibly closed by the remote host."
logger=grafana-apiserver t=2024-03-27T11:07:15.9188895-05:00 level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-03-27T11:07:15.9198844-05:00 level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
The service reports it failed to start, but not instantaneously like before. I’ll have to set the log level to debug for further troubleshooting.
which version of grafana did you download and install?
1 Like
The MSI installer does not create the path at \GrafanaLabs\grafana\data\plugins so that causes an error line.
I used the MSI, but marked the service install as not-install. It seems to hang trying to download NSSM.
What would the registry settings be if it were created via NSSM?
grafana-10.4.1.windows-amd64.msi is the installer I used. Thank you for helping today!
I am setting further reach-outs in the log, how can I disable these checks?
logger=plugin.signature.key_retriever t=2024-03-27T11:19:59.0069161-05:00 level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": read tcp 172.18.152.26:49659->34.120.177.193:443: wsarecv: An existing connection was forcibly closed by the remote host."
logger=grafana-apiserver t=2024-03-27T11:19:59.3949618-05:00 level=info msg="Adding GroupVersion playlist.grafana.app v0alpha1 to ResourceManager"
logger=grafana-apiserver t=2024-03-27T11:19:59.3959605-05:00 level=info msg="Adding GroupVersion featuretoggle.grafana.app v0alpha1 to ResourceManager"
logger=infra.usagestats t=2024-03-27T11:20:32.9860311-05:00 level=info msg="Usage stats are ready to report"
logger=plugin.signature.key_retriever t=2024-03-27T11:20:59.0242727-05:00 level=error msg="Error downloading plugin manifest keys" error="Get \"https://grafana.com/api/plugins/ci/keys\": read tcp 172.18.152.26:49677->34.120.177.193:443: wsarecv: An existing connection was forcibly closed by the remote host."
I would not go this route of modifying registry and stuff, sounds very hacky, I would leave that approach alone
just do what @isaqueprofeta said here Grafana-Server.exe vs Grafana.exe -Server?
Or download not the MSI but the zip version, extract then try using your powershell script to create the service. To disable those checks you would need to build grafana from code by removing those checks ( 
hole) If it does not prevent grafana from working let it go.
Unfortunately, in my knowledge of course, NSSM is the right way to go. I myself tried to do PowerShell Net-Service sometimes (not for Grafana), but it’s not the best way to do it since there’s a lot of uncovered features and options to make it work, and depends a lot on the software. As an example: Grafana’s binary doesn’t got any “stop” procedures, that’s one of the various things that should be considered.
I’d try to negotiate with my security team on using NSSM or, in a extreme, contact a good Microsoft Consulting on PowerShell to make it work right (I’d just use the binary without the service in this last case, honestly)
1 Like
If someone who has run NSSM for this can share this registry info, that could help me.
Get-ItemProperty -Path HKLM:\SYSTEM\CurrentControlSet\Services\Grafana\
Does Grafana generally run better on Linux? That’s about the same level of effort to get a clean new VM as it does to clear an application for the air-gapped environment.
Edit: Also want to note that when I switched from using LocalSystem to my own domain account with local admin access, Grafana-Server.exe does start long enough to start the main Grafana.exe. The service Grafana-Service does ultimately fail (hitting a timeout) but Grafana.exe stays up.
So air gapped implies absolutely no network connection. How were you able to use your domain creds?
Also why do you need the registry stuff curious
Just go for Linux. Better in every aspect for Grafana at least.
1 Like
Our air-gapped network is basically a wired LAN with no outside connections. It has its own Active Directory for authentication inside the network.
The reason I hope to see the registry info from a NSSM setup is to see if my settings are correct. Windows Services are just a few registry values.
@isaqueprofeta Net-Service is not PowerShell. New-Service is, I posted examples above.
Can we confirm that Grafana 10.x is compatible with Windows Server 2016?
I’m looking into Linux distros now. Thank you for your answers, it is appreciated.
1 Like
Yep it can run on 2016. But I would consider that overkill just for grafana unless you have tons of other things running on that server.
Go linux!
by the way grafana-server.exe does not seem to have Windows Service support embedded in it. When you use nssm it just running the exe as a fake service.
This is why it fails to start when using sc to install it or any other thing that assumes the exe has Windows Service support embedded in it
1 Like
The Windows Server 2016 has our SCCM/SCOM database on it, and has resources to spare.
That would be a wonderful feature to add! Grafana-Server.exe has other important command line parameters so it would be great to add it there.
Is it expected to have both Grafana-Server.exe running and also Grafana.exe?
