Grafana crashes when trying to connect to postgres: panic: setting PGSERVICEFILE not supported

danielt290947 · April 23, 2021, 3:31pm

postgres running and connectable, when setting up data source, grafana crashes
Hello, I have several grafana (free server version) running with postgres already.

Now, for the first time I have installed postgres in the automated way from our installation server. Obviously, there are many differences and it will be hardly possible to set up everything identical to the manual installation.
Up to now, setting up a postgres datasource was quite easy.
In the current installation, I can set it up. Then, when I click on “Save and Test”, Grafana crashes with following logging (see below, note: dots . substituted by underscore _, as I am not allowed to paste more than 2 “links”)

Grafana V. 7.3.1
Postgres 10.12
OS: SLES 12.5 (other installations are 12.4 or 12.2)

Note: I do not use a pg_service.conf file.
In the current installation I cannot find a pg_config

panic: setting PGSERVICEFILE not supported

goroutine 251 [running]:
github_com/lib/pq_parseEnviron_func2()
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/conn_go:1848
github_com/lib/pq_parseEnviron(0xc00014bc00, 0x62, 0x62, 0xc000b51aa0)
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/conn_go:1871 +0x1177
github_com/lib/pq_NewConnector(0xc00096eae0, 0x4c, 0xc000ab3798, 0xc000a82d80, 0xc000a82d80)
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/connector_go:56 +0x13e
github_com/lib/pq_DialOpen(0x220f540, 0xc000a82d80, 0xc00096eae0, 0x4c, 0x6e696f6a2e29297d, 0x652c292922202228, 0x6f6974636e75667d, 0x767b29742861206e)
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/conn_go:271 +0x39
github_com/lib/pq_Open()
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/conn_go:266
github_com/lib/pq_(*Driver)Open(0x33c1d78, 0xc00096eae0, 0x4c, 0x0, 0x0, 0x3d6f662072617600, 0x30212c2222280000)
/root/go/pkg/mod/github_com/lib/pq@v1_2_0/conn_go:48 +0x9a
database/sql_dsnConnector_Connect(_)
/usr/local/go/src/database/sql/sql_go:688
database/sql(*DB)conn(0xc000b2c300, 0x2223a40, 0xc0000ce010, 0x203001, 0x44747f, 0xc00005d800, 0x8)
/usr/local/go/src/database/sql/sql_go:1228 +0x201
database/sql(*DB)_query(0xc000b2c300, 0x2223a40, 0xc0000ce010, 0xc000698900, 0x35, 0x0, 0x0, 0x0, 0xc0005e0101, 0xc000072a80, __)
/usr/local/go/src/database/sql/sql_go:1565 +0x66
database/sql(*DB)_QueryContext(0xc000b2c300, 0x2223a40, 0xc0000ce010, 0xc000698900, 0x35, 0x0, 0x0, 0x0, 0xc000ab3b01, 0xc000b51a40, __)
/usr/local/go/src/database/sql/sql_go:1547 +0xd1
xorm_io/core(*DB)QueryContext(0xc0005e7b00, 0x2223a40, 0xc0000ce010, 0xc000698900, 0x35, 0x0, 0x0, 0x0, 0x0, 0x0, )
/root/go/pkg/mod/xorm_io/core@v0_7_3/db_go:121 +0x90
xorm_io/core(*DB)Query()
/root/go/pkg/mod/xorm_io/core@v0_7_3/db_go:133
github_com/grafana/grafana/pkg/tsdb/sqleng(*sqlQueryEndpoint)Query_func1(0xc0002186d0, 0xc000142000, 0xc0005e79c0, 0xc000698900, 0x35, 0xc0005e7a00, 0xc0005e1e80)
/root/project/pkg/tsdb/sqleng/sql_engine_go:169 +0x17d
created by github_com/grafana/grafana/pkg/tsdb/sqleng(*sqlQueryEndpoint)_Query
/root/project/pkg/tsdb/sqleng/sql_engine_go:163 +0x435

svetb · April 23, 2021, 3:42pm

Does this help (from pq package - github.com/lib/pq - Go Packages):

Most environment variables as specified at PostgreSQL: Documentation: 16: 34.15. Environment Variables supported by libpq are also supported by pq. If any of the environment variables not supported by pq are set, pq will panic during connection establishment. Environment variables have a lower precedence than explicitly provided connection parameters.

So if you (for some reason) have a PGSERVICEFILE environment variable set, you’ll need to remove it.

danielt290947 · April 26, 2021, 7:05am

Hello svetb,

thanks for the hint!

I have seen that page before. And, to tell the truth it rather confuses me, because all the env variable names there are only used there, I understand. When anybody wants to show any of the parameters, always the lowercase names are referred to, e.g. host instead of PGHOST. I was not able to find information how to check PGSERVICEFILE.
e.g. show PGSERVICEFILE; yield unrecognized parameter, the same with others.
In the shell, as postgres user, there is not any env set starting with PG.
What does Grafana differnetly from other software (jdbc on wildfly) and pgadmin, psql command line so that this problem occurs? Does Grafan try to set PGSERVICEFILE and how can I prevent it from?

svetb · April 26, 2021, 2:49pm

I don’t know enough about your setup to really comment on the source of the issue, but I did a quick experiment locally and can reproduce what you’re seeing, with the root cause I mentioned; namely:

If I set an environment variable PGSERVICEFILE=abcd (i.e. give it some random value), when I run Grafana and try to add a Postgres data source I get the same error
If I do not set this environment variable, everything works great

So you basically need to work out why that variable is set in Grafana’s environment. Note that this is not related to the environment that Postgres is running in, it’s the environment that the Grafana process is running in. And to your last question, no, it’s not Grafana itself that’s setting that environment variable, it’s something else on your system.

danielt290947 · April 27, 2021, 6:05am

Solved with Grafana running under user without PGSERVICEFILE
Thank you, that was the hint that brought me to the solution. 1st verified: The user under which Grafana was running has PGSERVICEFILE set. Started it with a user where PGSERVICEFILE is not set and it works.

danielt290947 · April 29, 2021, 12:45pm

A further hint for others: Just at that point in time I understood that the PG… variables are OS environment variables. That is why one cannot get them with select… or show… in pg. Just OS standard, e.g. echo §PGSERVICEFILE