postgresql: reorganize package and its extensions

[danbst]

Motivation for this change

Extracts some useful parts of #38698,
in particular, it's vision that postgresql plugins should be namespaced.

Original approach had several problems:

• not gonna happen in forseeable future
• did lots of deprecations
• was all-in-one solution, which is hard to sell to nixpkgs
• even that we have postgresqlPackages now, we can't do arbitrary overrides
  to postgresql and plugins. Several required functions were not exported

Here I've fixed all of those problems:

• deprecates nothing (though plugins were moved now into aliases.nix)
• this doesn't touch NixOS at all, and doesn't break anything
• hashes for plugins and PGs are not changed (I hope)
• no extra fixes to pg itself
• default PG is not changed
• plugins and PGs are extensible

Last one is the most interesting thing, because it introduces novel way
to manage XXX withPackages problem. It is novel, but has got lots of
inspiration from existing approaches:

• python, so we have now postgresql.pkgs.*, postgresql_11.pkgs.*
  which all contain plugins compiled with correct PG.
• python, so we have now postgresql.withPackages as well
• in contrast to python, there are no postgresql11Packages, only
  postgresql_11.pkgs
• Add pkgs.overrideWithScope #44196, so plugins are referenced starting at self-fixpoint.
  This allows override/add plugins with mere // in overlay. This works for
  both postgresqlPackages (overrides are applied to all postgresql_xx.pkgs)
  and postgresql_xx.pkgs (overrides are specific to this postgresql) sets
• I've made it compatible with proposed mergeable overlays (overlays: (partially) recursive merge #54266)
  however this PR doesn't depend on it
• last, but not least, postgresql/default.nix is now an overlay! This
  replaces previous callPackages approach with a modern, extensible concept.

Further improvements should go in subsequent PRs! I have a plan to merge @thoughtpolice 's PR part-by-part till 19.03. If nobody explicitly objects current implementation, I'll merge soon.

Things done

I've checked only that hashes didn't change for postgresql_9_6, 10, 11 and default plugins.

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Assured whether relevant documentation is up to date
• Fits CONTRIBUTING.md.

[danbst]

thanks, @ofborg

[danbst]

I'll show a few "override" example, for those who want to do that.

1. Update 1 plugin:

self: super: {
    postgresqlPackages = super.postgresqlPackages // {
        pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };
}

2. Create 2 PGs, one with plugin update, another with old version of plugin:

    postgresql_production = self.postgresql.withPackages (ps: with ps; [
        pg_repack
    ]);

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack-v20181024
    ]);

    postgresqlPackages = super.postgresqlPackages // {
        pg_repack-v20181024 = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };

3. Same, but lock plugins for one of postgresqls in such a way, that other overlays don't affect this. In fact, it is still possible to override, but becomes more tricky

    postgresql_production' = self.postgresql.override {
        this = self.postgresql_production';
        # 'extends' provide a pristine view of plugins, with whitelist overrides 
        postgresqlPackages = self.postgresql.pkgs.extend (_: plugins: {
            pg_similarity = plugins.pg_similarity.override { gcc = self.gcc8; };
        });
    };
    postgresql_production = self.postgresql_production'.withPackages (ps: with ps; [
        pg_repack  # default
        pg_similarity  # extended (overriden)
    ]);

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack  # overriden
        pg_similarity # default
    ]);

    postgresqlPackages = super.postgresqlPackages // {
        pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
            name = "pg_repack-v20181024";
            src = self.fetchzip {
                url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
            };
        });
    };

4. Override plugins only for postgresql 11

    postgresql_testing = self.postgresql.withPackages (ps: with ps; [
        pg_repack
        pg_similarity
    ]);

    postgresql_11 = super.postgresql_11.override { postgresqlPackages = self.postgresql_11.pkgs; } // {
        pkgs = super.postgresql_11.pkgs // {
            pg_repack = super.postgresqlPackages.pg_repack.overrideAttrs (_: {
                name = "pg_repack-v20181024";
                src = self.fetchzip {
                    url = "https://github.com/reorg/pg_repack/archive/923fa2f3c709a506e111cc963034bf2fd127aa00.tar.gz";
                    sha256 = "17k6hq9xaax87yz79j773qyigm4fwk8z4zh5cyp6z0sxnwfqxxw5";
                };
            });
        };
    };
    pg11_with_plugins = self.postgresql_11.withPackages (ps: with ps; [
        pg_repack
        pg_similarity
    ]);

@nbp does this look sane?
EDIT: these examples are no longer valid.

[ingenieroariel]

On this branch I tried both pkgs.postgresqlPackages.postgis and (postgis.override { postgresql = postgresql} ) and they failed.

The first one told me postgis did not exist and the second one gave me the following error:

pdfsdk_headers.h:53:10: fatal error: goo/gtypes.h: No such file or directory

On thoughtpolice's PR the first version works, on 18.09, the second version works.

[danbst]

@ingenieroariel that may be true. I did port only a little part here, and never touched postgis. See my plan #38698 (comment), so fixing postgis comes next after this change.

[danbst]

this has to be rebased over @thoughtpolice changes and fixes for postgis. Didn't expect problems, so developed yet another extender function

[danbst]

Removed all my "novelties" and used plain overlays instead. No more fancy "override with default", but at least overrides in overlays work predictably. Ability to override with // is preserved, and it would be even more awesome after #44196 and #54266

[nixos-discourse]

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/heads-up-upcoming-changes-to-the-postgresql-infrastructure-and-nixos-module/297/5