Skip to content

[WIP]: PostgreSQL declarative databases #72365

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
talyz wants to merge 2 commits into from
Closed

[WIP]: PostgreSQL declarative databases #72365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
55dad40
nixos/postgresql: Add ability to specify db extensions and owner
talyz Aug 16, 2019
4f8c25a
nixos/postgresql: Drop databases when removed from ensureDatabases
talyz Oct 28, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 139 additions & 47 deletions nixos/modules/services/databases/postgresql.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,86 @@ in

###### interface

options = {

options = let
databaseOpts = {
options = {
name = mkOption {
type = types.str;
description = ''
The name of the database to ensure.
'';
example = "gitlab";
};
owner = mkOption {
type = with types; nullOr str;
default = null;
description = ''
The name of the database owner.

The user will be created if it doesn't already
exist. The owner of an existing database will be
changed if it's changed here, and all objects in the
database owned by the previous owner will be
reassigned to the new one.

See
<link xlink:href="https://www.postgresql.org/docs/current/manage-ag-createdb.html">Creating a Database</link>,
<link xlink:href="https://www.postgresql.org/docs/current/sql-reassign-owned.html">REASSIGN OWNED</link>
and
<link xlink:href="https://www.postgresql.org/docs/current/sql-alterdatabase.html">ALTER DATABASE</link>
for more details.
'';
};
extensions = mkOption {
Copy link
Contributor

@kampka kampka Dec 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option seems to be on the wrong level.
Extensions in Postgres do not exist on a database level.
In fact, you cannot even have the same extension loaded in different versions on different schemas, even though the documentation might suggest that.
Because of that, I think this option belongs on the service layer, not on a database specific layer in the options.

Copy link
Contributor Author

@talyz talyz Dec 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, that doesn't sound right. I'd have to test this myself to confirm, but if that's true, the documentation is plain wrong. Refer to https://www.postgresql.org/docs/current/sql-createextension.html - the first sentence is CREATE EXTENSION loads a new extension into the current database.

Copy link
Contributor

@kampka kampka Dec 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not impossible I am remembering this from an older version, but I have this nagging feeling that something was iffy here. Best to test, I agree.

type = with types; listOf str;
default = [];
description = ''
Extensions to add to the database.

Extensions will be added to existing databases, but
not removed if they're removed from the list.

See
<link xlink:href="https://www.postgresql.org/docs/current/sql-createextension.html">
CREATE EXTENSION</link> for more details.
'';
example = [ "pg_trgm" "hstore" ];
};
};
};
userOpts = {
options = {
name = mkOption {
type = types.str;
description = ''
Name of the user to ensure.
'';
};
ensurePermissions = mkOption {
type = types.attrsOf types.str;
default = {};
description = ''
Permissions to ensure for the user, specified as an attribute set.
The attribute names specify the database and tables to grant the permissions for.
The attribute values specify the permissions to grant. You may specify one or
multiple comma-separated SQL privileges here.

For more information on how to specify the target
and on which privileges exist, see the
<link xlink:href="https://www.postgresql.org/docs/current/sql-grant.html">GRANT syntax</link>.
The attributes are used as <code>GRANT ''${attrName} ON ''${attrValue}</code>.
'';
example = literalExample ''
{
"DATABASE nextcloud" = "ALL PRIVILEGES";
"ALL TABLES IN SCHEMA public" = "ALL PRIVILEGES";
}
'';
};
};
};
in
{
services.postgresql = {

enable = mkOption {
Expand Down Expand Up @@ -97,13 +175,23 @@ in
};

ensureDatabases = mkOption {
type = types.listOf types.str;
type = with types;
listOf (coercedTo str
(name: { inherit name; })
(submodule databaseOpts));
default = [];
description = ''
Ensures that the specified databases exist.
This option will never delete existing databases, especially not when the value of this
option is changed. This means that databases created once through this option or
otherwise have to be removed manually.
Ensure the existence of the listed databases with their
respective owner and extensions.

This option will never delete existing databases; if the
Copy link
Contributor

@florianjacob florianjacob Nov 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs to be adjusted

name of a database is changed here, a new database with that
name will be created and the old one will remain. Databases
created once through this option or otherwise have to be
removed manually.

Extensions will be added to existing databases, but not
removed if they're removed from the list.
'';
example = [
"gitea"
Expand All @@ -112,37 +200,7 @@ in
};

ensureUsers = mkOption {
type = types.listOf (types.submodule {
options = {
name = mkOption {
type = types.str;
description = ''
Name of the user to ensure.
'';
};
ensurePermissions = mkOption {
type = types.attrsOf types.str;
default = {};
description = ''
Permissions to ensure for the user, specified as an attribute set.
The attribute names specify the database and tables to grant the permissions for.
The attribute values specify the permissions to grant. You may specify one or
multiple comma-separated SQL privileges here.

For more information on how to specify the target
and on which privileges exist, see the
<link xlink:href="https://www.postgresql.org/docs/current/sql-grant.html">GRANT syntax</link>.
The attributes are used as <code>GRANT ''${attrName} ON ''${attrValue}</code>.
'';
example = literalExample ''
{
"DATABASE nextcloud" = "ALL PRIVILEGES";
"ALL TABLES IN SCHEMA public" = "ALL PRIVILEGES";
}
'';
};
};
});
type = with types; listOf (submodule userOpts);
default = [];
description = ''
Ensures that the specified users exist and have at least the ensured permissions.
Expand Down Expand Up @@ -320,6 +378,7 @@ in
# Wait for PostgreSQL to be ready to accept connections.
postStart =
''
set -eu
PSQL="${pkgs.sudo}/bin/sudo -u ${cfg.superUser} psql --port=${toString cfg.port}"

while ! $PSQL -d postgres -c "" 2> /dev/null; do
Expand All @@ -333,18 +392,51 @@ in
''}
rm -f "${cfg.dataDir}/.first_startup"
fi
'' + optionalString (cfg.ensureDatabases != []) ''
${concatMapStrings (database: ''
$PSQL -tAc "SELECT 1 FROM pg_database WHERE datname = '${database}'" | grep -q 1 || $PSQL -tAc 'CREATE DATABASE "${database}"'
'') cfg.ensureDatabases}
'' + ''
${concatMapStrings (user: ''
$PSQL -tAc "SELECT 1 FROM pg_roles WHERE rolname='${user.name}'" | grep -q 1 || $PSQL -tAc "CREATE USER ${user.name}"
''
+ concatMapStrings (user: ''
$PSQL -tAc "SELECT 1 FROM pg_roles WHERE rolname='${user}'" | grep -q 1 || $PSQL -tAc "CREATE USER ${user}"
'') (unique (concatMap (a: if a ? owner then if a.owner != null then [a.owner] else [] else [a.name]) (cfg.ensureUsers ++ cfg.ensureDatabases)))
+ concatMapStrings (db: ''
database_exists=$($PSQL -tAc "SELECT 'true' FROM pg_database WHERE datname = '${db.name}'")
if [[ "$database_exists" != "true" ]]; then
$PSQL -tAc 'CREATE DATABASE "${db.name}" ${optionalString (db.owner != null) ''OWNER "${db.owner}"''}'
fi

echo "${db.name}" >> "${cfg.dataDir}/databases"

${optionalString (db.owner != null) ''
current_owner=$($PSQL -tAc "SELECT pg_catalog.pg_get_userbyid(datdba) FROM pg_catalog.pg_database WHERE datname = '${db.name}'")
if [[ "$current_owner" != "${db.owner}" ]]; then
$PSQL -tAc 'ALTER DATABASE "${db.name}" OWNER TO "${db.owner}"'
if [[ -e "${cfg.dataDir}/.reassigning_${db.name}" ]]; then
echo "Reassigning ownership of ${db.name} to ${db.owner} failed on last boot. Failing..."
exit 1
fi
touch "${cfg.dataDir}/.reassigning_${db.name}"
$PSQL "${db.name}" -tAc "REASSIGN OWNED BY \"$current_owner\" TO \"${db.owner}\""
rm "${cfg.dataDir}/.reassigning_${db.name}"
fi
''}
''
+ concatMapStrings (extension: ''
$PSQL '${db.name}' -tAc "CREATE EXTENSION IF NOT EXISTS ${extension}"
'') db.extensions
) cfg.ensureDatabases
+ ''
touch "${cfg.dataDir}/databases"
if [[ -f "${cfg.dataDir}/existing_databases" ]]; then
for db in $(grep -vFxf "${cfg.dataDir}/databases" "${cfg.dataDir}/existing_databases" || echo ""); do
echo "Dropping database $db!"
$PSQL -tAc "DROP DATABASE \"$db\""
done
fi
mv "${cfg.dataDir}/databases" "${cfg.dataDir}/existing_databases"
''
+ concatMapStrings (user: ''
${concatStringsSep "\n" (mapAttrsToList (database: permission: ''
$PSQL -tAc 'GRANT ${permission} ON ${database} TO ${user.name}'
'') user.ensurePermissions)}
'') cfg.ensureUsers}
'';
'') cfg.ensureUsers;

unitConfig.RequiresMountsFor = "${cfg.dataDir}";
};
Expand Down
26 changes: 5 additions & 21 deletions nixos/modules/services/misc/gitlab.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -602,28 +602,12 @@ in {
# We use postgres as the main data store.
services.postgresql = optionalAttrs databaseActuallyCreateLocally {
enable = true;
ensureUsers = singleton { name = cfg.databaseUsername; };
ensureDatabases = singleton {
name = cfg.databaseName;
owner = cfg.databaseUsername;
extensions = [ "pg_trgm" ];
};
};
# The postgresql module doesn't currently support concepts like
# objects owners and extensions; for now we tack on what's needed
# here.
systemd.services.postgresql.postStart = mkAfter (optionalString databaseActuallyCreateLocally ''
set -eu

$PSQL -tAc "SELECT 1 FROM pg_database WHERE datname = '${cfg.databaseName}'" | grep -q 1 || $PSQL -tAc 'CREATE DATABASE "${cfg.databaseName}" OWNER "${cfg.databaseUsername}"'
current_owner=$($PSQL -tAc "SELECT pg_catalog.pg_get_userbyid(datdba) FROM pg_catalog.pg_database WHERE datname = '${cfg.databaseName}'")
if [[ "$current_owner" != "${cfg.databaseUsername}" ]]; then
$PSQL -tAc 'ALTER DATABASE "${cfg.databaseName}" OWNER TO "${cfg.databaseUsername}"'
if [[ -e "${config.services.postgresql.dataDir}/.reassigning_${cfg.databaseName}" ]]; then
echo "Reassigning ownership of database ${cfg.databaseName} to user ${cfg.databaseUsername} failed on last boot. Failing..."
exit 1
fi
touch "${config.services.postgresql.dataDir}/.reassigning_${cfg.databaseName}"
$PSQL "${cfg.databaseName}" -tAc "REASSIGN OWNED BY \"$current_owner\" TO \"${cfg.databaseUsername}\""
rm "${config.services.postgresql.dataDir}/.reassigning_${cfg.databaseName}"
fi
$PSQL '${cfg.databaseName}' -tAc "CREATE EXTENSION IF NOT EXISTS pg_trgm"
'');

# Use postfix to send out mails.
services.postfix.enable = mkDefault true;
Expand Down