Trouble getting pgscv running

[greigwise]

I'm getting a segfault trying to run pgscv on a pgbouncer instance, connecting remotely to my postgres server. If I disable the postgres/settings collector, then I can avoid the segfault. Wondering if maybe it's because I should be running pgscv on the postgres server itself? Even if that is right, I wouldn't expect a segfault. Here is what I'm seeing:

Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: {"level":"warn","service":"pgscv","time":"2021-08-18T18:23:18Z","message":"stat /var/lib/pgsql/12/data/postgresql.conf failed: stat /var/lib/pgsql/12/data/postgresql.conf: no such file or directory; skip"}
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: panic: runtime error: invalid memory address or nil pointer dereference
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xa2fa88]
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: goroutine 50 [running]:
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: github.com/weaponry/pgscv/internal/collector.parsePostgresFiles(0xc000250240, 0xc17bba, 0x77, 0xc000250240)
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: /home/runner/work/pgscv/pgscv/internal/collector/postgres_settings.go:227 +0x1c8
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: github.com/weaponry/pgscv/internal/collector.(*postgresSettingsCollector).Update(0xc0000c2a50, 0xc0000356e8, 0x8, 0xc0000ac270, 0x28, 0x0, 0x0, 0x0, 0x1, 0x2000, ...)
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: /home/runner/work/pgscv/pgscv/internal/collector/postgres_settings.go:77 +0x305
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: github.com/weaponry/pgscv/internal/collector.collect(0xbf70b9, 0x11, 0xc0000356e8, 0x8, 0xc0000ac270, 0x28, 0x0, 0x0, 0x0, 0x1, ...)
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: /home/runner/work/pgscv/pgscv/internal/collector/collector.go:241 +0x85
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: github.com/weaponry/pgscv/internal/collector.PgscvCollector.Collect.func1(0xc000120a00, 0xc00006a360, 0xc0004f8890, 0xbf70b9, 0x11, 0xcc7300, 0xc0000c2a50)
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: /home/runner/work/pgscv/pgscv/internal/collector/collector.go:205 +0xc8
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: created by github.com/weaponry/pgscv/internal/collector.PgscvCollector.Collect
Aug 18 18:23:18 pool01-qa-uw2.hireology.com pgscv[8348]: /home/runner/work/pgscv/pgscv/internal/collector/collector.go:204 +0x1ed
Aug 18 18:23:18 pool01-qa-uw2.hireology.com systemd[1]: pgscv.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Aug 18 18:23:18 pool01-qa-uw2.hireology.com systemd[1]: Unit pgscv.service entered failed state.
Aug 18 18:23:18 pool01-qa-uw2.hireology.com systemd[1]: pgscv.service failed.

Also wondering why I'm getting the errors posted below. Maybe also because I'm not running on the postgres server itself?

Aug 18 18:30:18 pool01-qa-uw2.hireology.com pgscv[11322]: {"level":"error","service":"pgscv","time":"2021-08-18T18:30:18Z","message":"find tablespaces mountpoint failed: lstat /var/lib/pgsql/12/data: no such file or directory"}
Aug 18 18:30:18 pool01-qa-uw2.hireology.com pgscv[11322]: {"level":"error","service":"pgscv","time":"2021-08-18T18:30:18Z","message":"get WAL directory size failed: conn busy"}
Aug 18 18:30:18 pool01-qa-uw2.hireology.com pgscv[11322]: {"level":"error","service":"pgscv","time":"2021-08-18T18:30:18Z","message":"get log directory size failed: conn busy"}
Aug 18 18:30:18 pool01-qa-uw2.hireology.com pgscv[11322]: {"level":"error","service":"pgscv","time":"2021-08-18T18:30:18Z","message":"get total size of temp files failed: conn busy"}

Thanks for the help,
Greig Wise

[lesovsky]

Hi Greg,
for any kind of such errors (like segfaults) it is better to create an issue.

Could you provide a yaml config used?

[greigwise]

I will create an issue and post the config file there. With regards to my second question about the errors finding tablespaces, getting WAL directory size, etc., do you have any suggestions?

Thanks,
Greig Wise

[lesovsky]

Thanks for the issue.

About the second question, I am not ready to ask right now, need to make some tests with your config and environment similar to yours.

[lesovsky]

Hi Greig,

I've reviewed the attached YAML config from the issue. The problem is that, you connecting to Postgres through the local Pgbouncer (like a proxy). In such case Postgres service considered as running locally and all collectors are enabled (but in fact Postgres is located on remote system). Enabled postgres/storage collector thinks the Postgres is running locally and fails, because it requires local access to Postgres data directory.

The problem could be resolved by specifying direct connection to Postgres service.