Flarum

[fricklerhandwerk]

Flarum is a technically advanced, open and extensible discussion platform. Flarum aims to bring people interaction to a new level by how it is designed and engineered.

• NLnet project
• source code

This project is very much alive, with a large user base, and not packaged anywhere. Providing a turnkey service module for self-hosting is a high-value target.

[albertchae]

@ngi-nix/moss is claiming this

[jasonodoom]

We used composer2nix on the output of composer create-project flarum/flarum. for installing flarum because composer.lock isn't checked into the flarum repo.

We need to find a way to include composer.lock into the sources. How do we merge sources? Also, what are the quality expectations in regards to code style?

[fricklerhandwerk]

Not sure why you would need to merge sources for that. composer2nix has to be run manually, otherwise you'll have IFD.

What you may want instead – but I could be completely wrong – is a helper script in the devshell that checks out the source into a tempdir, runs composer create-project flarum/flarum ., runs composer2nix on the output, and overwrites the generated Nix expressions in this repo. And of course some documentation what to do when and why. I don't know what the code generated by composer2nix looks like, please open a PR with what you have so far if you need more input.

[albertchae]

@fricklerhandwerk I opened a PR with our WIP branch so far. The line in particular that is making us think we need to merge sources is #23 (comment)

What you may want instead – but I could be completely wrong – is a helper script in the devshell that checks out the source into a tempdir, runs composer create-project flarum/flarum ., runs composer2nix on the output, and overwrites the generated Nix expressions in this repo. And of course some documentation what to do when and why.

Sorry if I'm misunderstanding, but is the helper script you're suggesting something we'd run manually when we need to update pkgs/flarum in this repo? Or is it something that'd run as part of building the flarum package?

[albertchae]

@fricklerhandwerk Oh also, do you have a contact at the flarum project in case we have additional questions or do you recommend just opening an issue on their repo? No one in our mob is that familiar with the php ecosystem and their best practices, so being able to get more context would be helpful.

[fricklerhandwerk]

is the helper script you're suggesting something we'd run manually when we need to update pkgs/flarum in this repo? Or is it something that'd run as part of building the flarum package?

Yes, it would have to be run manually. As said, it will be IFD if you make it part of the build, because you'll first have to realise the derivation with the source definitions before you can continue evaluating the actual package definition. I just assume we don't want IFD in a monorepo, as the repo's expression may grow arbitrarily large and we'd quickly run into issues with evaluation time. Right, @cleeyv?

do you have a contact at the flarum project in case we have additional questions or do you recommend just opening an issue on their repo?

No. I suggest getting in touch with developers directly and only then open issues. We don't know if they are interested in collaboration and don't want to force Nix on them if they don't care. I did not manage to contact everyone, but I'll continue collecting contextual information for you when I get to it.

[albertchae]

Disclaimer: braindump for @ngi-nix/moss , please ignore if you're not in @ngi-nix/moss

Notes from July 24 mob session:

• We found a closed PR from several years ago on nixpkgs that attempted to package flarum and saw what we could learn from it flarum, nixos/flarum: init at 0.1.0-beta.13 NixOS/nixpkgs#96869

  • One thing they did is distinguish between modules and packages. It probably makes sense to make a module for flarum because we need to configure nginx and other runtime dependencies.
  • Their filesystem hierarchy
  • They commit composer.json and composer.lock and use them while packaging flarum
  • We weren't able to nix build .#flarum when checking their branch.

• Our next idea was to identify success examples of using composer2nix to add packages to nixpkgs. We found:

  • pixelfed
  • bookstack
  • snipe-it
  • All of these packages are almost the same except for the shell commands in the install phase of default.nix.
    • Side note: These packages include an update.sh which seems to be used to make bumping the composer.lock and version number more convenient. We are choosing to ignore this for now.

• We were able to copy this structure and get to the point where we got an error with composer.json not matching composer.lock

       > - Required package "flarum/nicknames" is not present in the lock file.                               
       > - Required package "flarum/pusher" is not present in the lock file.                                  
       > - Required package "flarum/statistics" is not present in the lock file.                              
       > - Required package "flarum/sticky" is not present in the lock file.                                  
       > - Required package "flarum/subscriptions" is not present in the lock file.                           
       > - Required package "flarum/suspend" is not present in the lock file.                                 
       > - Required package "flarum/tags" is not present in the lock file.                                    
       > This usually happens when composer files are incorrectly merged or the composer.json file is manually edited.
  

• We believe this is happening because the flarum/flarum repo does not have a composer.lock checked in and composer2nix generates a stub composer.lock

• We obtained a composer.lock by running composer create-project flarum/flarum . as recommended by https://docs.flarum.org/install/#installing. We also tested that the generated composer.lock is the equivalent of running composer install in a checkout of flarum/flarum. (We also tried deleting the vendor directory before running composer2nix to make the process pure, which produced the same file with no warning.

• composer.lock seems to be intentionally git ignored and not checked into flarum/flarum

  • flarum/flarum@995a262
    • This appears to be the reason https://blog.martinhujer.cz/17-tips-for-using-composer-efficiently/#tip-%236%3A-put-%60composer.lock%60-into-%60.gitignore%60-in-libraries
  • flarum/flarum@ea61c22

• Our conclusion so far is that users are using flarum/flarum as a template by creating a project based on it, and then committing the lock file on their created project. This seems to be a supported pattern https://getcomposer.org/doc/03-cli.md#create-project. It seems composer2nix does not support projects with missing composer.lock files (see issue). So we should either modify composer2nix to handle that or modify the generated code or add something later to overlay (not necessarily the nix overlay) our composer.lock on top of flarum/flarum.

[cleeyv]

I just assume we don't want IFD in a monorepo, as the repo's expression may grow arbitrarily large and we'd quickly run into issues with evaluation time. Right, @cleeyv?

Yes, my work on the CI is quickly surfacing problems with IFDs since there is already one there for mach-nix in the liberaforms package. There are evaluation-time performance impacts, as you say, but also other problems (explained in #24). I understand why nixpkgs doesn't allow them and I think it could be prudent to consider this rule for ngipkgs as well.

@albertchae while you're considering options for packaging Flarum, you may want to check out this PR on nixpkgs for a new builder for PHP Composer-based applications: NixOS/nixpkgs#225401 It is marked as a draft, but reading the comments it sounds like it is pretty late in the process, and ready to be tested. It would be cool if you could use Flarum to do a test of this the builder and give feedback for the PR review process. And even if you don't test it out, the comments on the PR seem like they could provide relevant background info for your work on the Flarum packaging.

[albertchae]

@cleeyv Thanks! Coincidentally we also came across NixOS/nixpkgs#225401 right after we wrote our brain dump, so trying that out will probably be the focus of our remaining mob sessions this week.

[DMills27]

Disclaimer: braindump for https://github.com/orgs/ngi-nix/teams/moss , please ignore if you're not in https://github.com/orgs/ngi-nix/teams/moss

After realising that composer2nix doesn't work without a composer.lock file, we decided upon the builder from this PR. We created a second branch moss-flarum-2 to try this approach. This has a number of .nix files that can be found in this commit. We had to modify namespaces in order to get it running, more specifically phpDrvand php in order to get it to build. Also, in this PR the callPackages function was defined in the same way and we're currently not sure why it worked but it did.

We followed the pattern of wp-cli and others, where a composer.lock file was generated and checked into the Nix build directory and then successfully built the flarum package.

[cleeyv]

Cross-posted from a parallel conversation over in issue #24:

So we haven't explicitly been testing whether flarum is compatible with darwin (but also there's no reason it shouldn't be compatible? we just haven't tried).

Based on a quick look through the flarum documentation, I get the impression that in theory it could be run on MacOS but since that server hardware is relatively rare/expensive, practically it is much more likely to be run in production on linux. However, since MacOS is way more common for dev machines, they do mention in their section on Development Tools the use of tools like Laravel Valet to do local dev on Flarum. Based on this, I can imagine the usefulness of a dev shell / local deployment config of some kind for darwin. Though it also sounds like you've already been supporting that use case on x86_64-linux with lima which maybe is simpler. I'm curious to hear if you have further thoughts on this (I'm going to cross-post this over to the flarum issue as well).

[jleightcap]

Disclaimer: braindump for https://github.com/orgs/ngi-nix/teams/moss , please ignore if you're not in https://github.com/orgs/ngi-nix/teams/moss

Moss mob status update for 2023-07-27

• we started the session by beginning work on defining the flarum module. Since the module is independent of how the flarum package is defined (as long as ngipkgs.flarum works), this work isn't blocked by the composer2nix/buildComposerProject work.
  • we began by wholesale copying the extant flarum module definition here.
  • this had some incompatible definitions (pkgs vs ngipkgs &c), and seems to have some questionable configuration file setup (toJSON is used to generate a config.yml? And used as flarum install --file=config.yml which seems undocumented, or at least as a result of the closed PR targeting an older release of flarum.)
• with a module defined, we needed some was of testing/verifying;
  • since ngipkgs doesn't seem to include a testing framework (related? Support for NixOS tests #28) e.g. home-manager, we decided to enable the service on a live NixOS system
  • (lots of flailing with a morph DigitalOcean image that ended up at a dead end because of flakes incompatibility)
  • running on @Chickensoupwithrice's bare-metal machine, we ran into:

   at /nix/store/b1vy558z7lxph5mbg7n50b5njp393ia9-source/lib/modules.nix:483:28:

      482|         builtins.addErrorContext (context name)
      483|           (args.${name} or config._module.args.${name})
         |                            ^
      484|       ) (lib.functionArgs f);

• we decided to reference the existing module (liberaforms) defined in ngipkgs, which when initialized the same way, fails with the same error.

          modules = [
            # ... 
            ngipkgs.nixosModules.x86_64-linux.modules.liberaforms
            ({ pkgs, ... }:
              {
                environment.systemPackages = [ inputs.ngipkgs.packages.x86_64-linux.flarum ]; # <- this works!
                services.liberaforms.enable = true; # <- this yields the error above
              }
           ]

• so, we're hit a blocker that either that we're initializing modules in general incorrectly, or the liberaform module is incomplete the same way as flarum.
  • we're erring on... we're probably just importing this modules incorrectly.
  • is a separate repository for flarum correct? following the example of https://github.com/ngi-nix/liberaforms-flake (although we may not understand the context behind this repo.)
  • since this issue seems to be beyond purely packaging ngipkgs, this would be great to feedback and have documented.

[cleeyv]

since ngipkgs doesn't seem to include a testing framework (related? #28) e.g. home-manager, we decided to enable the service on a live NixOS system

Yes, seeing if we can get NixOS tests working on our hydra instance is one of the top priorities for further improvement to ngipkgs. The timing is unfortunate that I'm going to be away for the next two weeks so won't be able to work on it soon, but there should be progress on this later in August.

(lots of flailing with a morph DigitalOcean image that ended up at a dead end because of flakes incompatibility)

When I was a Summer of Nix participant in past years, we did this kind of flake deployment to digital ocean using deploy-rs and it worked pretty well. The main difference is that at that time each project had its own repo and flake, so a similar use of deploy-rs would still need to be tested on ngipkgs. I created an issue for work on this: #22.

we decided to reference the existing module (liberaforms) defined in ngipkgs, which when initialized the same way, fails with the same error.

When it was in its own liberaforms-flake repo, we deployed liberaforms in two ways: with deploy-rs to digital ocean, and with nixos-container to a local container. On the old repo, these two methods are documented here: https://github.com/ngi-nix/liberaforms-flake/tree/main/docs

Since the migration to ngipkgs, so far we have only implemented deployment with nixos-container, and that is confirmed to be working. Well, there is currently some error with liberaforms when deployed to a local container (the web app doesn't properly load in the browser), but it is an error page that is generated by liberaforms so we know the build and deployment and system configuration aspects are all working at least.

An advantage of trying to do the same kind of nixos-container deployment with flarum as well is that you could more directly model it on the currently working liberaforms example in ngipkgs.

is a separate repository for flarum correct? following the example of https://github.com/ngi-nix/liberaforms-flake (although we may not understand the context behind this repo.)

A separate repository for each project was the old way things were organized for nix packaging at NGI, and for Summer of Nix. This year it was decided to consolidate all of the packaging work into a single monorepo, ngipkgs. Your use of it here for flarum is experimental, and we're all learning along the way. As the the program progresses, these questions of how to use ngipkgs will become more clear.

[jleightcap]

Yes, seeing if we can get NixOS tests working on our hydra instance is one of the top priorities for further improvement to ngipkgs. The timing is unfortunate that I'm going to be away for the next two weeks so won't be able to work on it soon, but there should be progress on this later in August.

awesome!

When I was a Summer of Nix participant in past years, we did this kind of flake deployment to digital ocean using deploy-rs and it worked pretty well. The main difference is that at that time each project had its own repo and flake, so a similar use of deploy-rs would still need to be tested on ngipkgs. I created an issue for work on this: #22.

thanks for the pointer; based on https://github.com/ngi-nix/deploy-liberaforms i've migrated our shared DO deployment to use `deploy-rs` rather than the flakes-incompatible `morph`. this live system should be enough if we eventually run a non-containerized instance of Flarum.

When it was in its own liberaforms-flake repo, we deployed liberaforms in two ways: with deploy-rs to digital ocean, and with nixos-container to a local container. On the old repo, these two methods are documented here: https://github.com/ngi-nix/liberaforms-flake/tree/main/docs Since the migration to ngipkgs, so far we have only implemented deployment with nixos-container, and that is confirmed to be working. Well, there is currently some error with liberaforms when deployed to a local container (the web app doesn't properly load in the browser), but it is an error page that is generated by liberaforms so we know the build and deployment and system configuration aspects are all working at least. An advantage of trying to do the same kind of nixos-container deployment with flarum as well is that you could more directly model it on the currently working liberaforms example in ngipkgs.

ah yeah, the nixos-container documentation in the README should be a starting point for a similar smoke test with Flarum. RTFM :)

A separate repository for each project was the old way things were organized for nix packaging at NGI, and for Summer of Nix. This year it was decided to consolidate all of the packaging work into a single monorepo, ngipkgs. Your use of it here for flarum is experimental, and we're all learning along the way. As the the program progresses, these questions of how to use ngipkgs will become more clear.

we found #12 shortly after which provided more historical context to the in-progress monorepo migration. we'll document our progress with the module development for others packaging modules here.

[jleightcap]

Note: the Flarum module seems to revolve around a root config.php, which doesn't have a native Nix builder. Some package in upstream get around this by proxying through JSON: https://github.com/NixOS/nixpkgs/blob/6dd8d1ffc7e24340c02ca89bbddc3b24b192352a/nixos/modules/services/web-apps/limesurvey.nix#L21-L25

but for the moment we're attempting to get working Flarum instance using a raw string.

[jleightcap]

Putting this on pause for the moment, with the package finished we're waiting for direction on module definitions and testing.

[cleeyv]

I see that moss is working on this again in https://github.com/ngi-nix/ngipkgs/tree/moss-flarum-module-rebased-on-pretalx , that you were previously trying to do testing with a deploy-rs to a cloud instance (as discussed in #22), and that algae has shared their local testing approach using nixos-test (#28). I will just add that nixos-container is another option that can be used for a similar kind of local testing and the configs/liberaforms/containers.nix is available as a template. If you have any questions about this approach to local testing, let me know.

[Chickensoupwithrice]

We're working on getting the Flarum module running.
Currently we decided to have nginx own the directory from which flarum is run. In order to set this up we have a systemd oneshot script. Previously, the users.user config was creating a flarum user, and setting it's home directory to be cfg.stateDir which had the side effect of creating this directory with flarum permissions.
Now we want to create this directory with nginx permissions, however when we removed the users.user config option, the oneshot script no longer executes at all.
One very crude solution to this could be running the systemd script as user root, and having that chown the directory. We would much prefer that the systemd oneshot script (which is being run as nginx) also create the directory (using the stateDirectory config). When manually run, the systemd unit does what we expect, but we're unable to get it to start on boot/login

[lorenzleutgeb]

FWIW, instead of fiddling around with permissions explicitly, consider the following options:

• RuntimeDirectory=, StateDirectory=, CacheDirectory=, LogsDirectory=, ConfigurationDirectory=
• DynamicUser=
• Capabilities
• Credentials

[albertchae]

Turns out #10 (comment) was a red herring due to the install oneshot service running multiple times due to downstream services failing and retrying

[albertchae]

flarum has been upstreamed to nixpkgs in NixOS/nixpkgs#311608 so it should also be a candidate for #76