Remove generated content from being tracked in Git

[tunetheweb]

Fixes #1221 by removing all generated content from git, adding .gitignore entries to stop it getting added back, and adding deploy script to ensure it's generated before deploying.

I’ve left the PDFs in there as unlikely to be confused for real content and allows us to continue to use the GitHub Action to generate the ebooks rather than make people have to install Prince locally.

[tunetheweb]

@rviscomi I think I've addressed all your feedback and made some improvements too. Could you have another look?

@ibnesayeed I'd value your input here too!

[ibnesayeed]

One way to envision the whole production branch is as following:

• Track only the source files in the main branch as the source of truth that can be used to generate the site at any historical state
• Add a build process that creates a dist folder that contains the complete directory structure of only files needed to serve the site, such as:
  • All the generated files
  • Scripts that are needed at run time, but no other helper scripts to maintain the project
  • Derived files such as bundled JS/CSS (minified or otherwise), but not the pre-transpilation files such as CoffeeScript/TypeScript/SASS
  • Files that are going to be the same in both the source as well as the runtime environment
• Exclude the dist folder using the .gitignore file
• Force push the content of the dist folder to the production branch
• Additionally, if maintaining the history of the production branch is not important, it can be purged each time before pushing new content to it to only contain the most recent distribution

[tunetheweb]

That’s kind of what I’m going for here @ibnesayeed but in a slightly different manner than you’re suggesting.

It’s not quite as easy to separate source code from deployed (dist) code as you are suggesting.

For a start some of our source code is deployed as is. Particularly our Python code, the static folder and most of the templates. So we’d duplicate that code in the deploy folder, which seems a little pointless - why not keep the folder structure we have and use the .gcloudignore like I’m suggesting?

Also some of our helper scripts (particularly those in the tools/generate folder) are important to see how and why the artifacts were built the way they were. We are not using the release to actually deploy, nor to feed downstream (can’t see anyone becoming dependent on us - but who knows what the future holds!). So I see the release artifact as the snapshot in time as to what was released in case we even need to go back to understand what exactly was in a release for whatever reason. Yes in theory the git tagging does that too, but as there is generated content that will no longer be tracked due to this PR I think it’s important to track what was deployed - not just the source files. Which is the point of that. Maybe the deployer had a different machine setup and so that might explain why the deployed files were as they should have been? Or maybe they didn’t have a clean checkout? So I think having the source files, the build files, and the generated content all in the deploy.zip file will be helpful in case we were need to do some investigation to identify a release issue.

I also don’t know how you can force push a folder that’s in .gitignore - is that possible?

I also dislike the idea of production diverging from main. That way lies merge conflicts and errors IMHO. And while it’s true that production is its own branch, there is nothing to stop is throwing it away and resetting or rebasing to main if it gets in a mess. That’s not possible if it diverges - at least not cleanly without losing history. While the most benefit it probably in the latest deploy as you say, why not use GitHub to track them all?

I think storing the released artifacts in the Release section of GitHub is a lot cleaner, gives us the history, and links it to a tagged version in Git so really like that workflow. Plus it gives us a release history and allows other contributors to watch for releases to see when site changes (maybe when their development, or writing, or translation finally goes live!).

Plus for me, the Draft release notes let’s me see when items are piling up and we should cut a release, as well as reminding me what exactly has changed since the last release.

[tunetheweb]

Some observations after the first manual deploy of the site using this process:

Zipping the assets produced a 91MB zip file. That’s a bit bigger than I realised it would be (It’s mostly the images) and it does take a minute or two to upload on my connection.

It looks like GitHub automatically adds the state of the repo as assets at the bottom of a release (in both zip and tar.gz formats). That contains 90% of what we need and there’s a lot of duplication.

We could either just zip up the generated content, in the script in this PR, to make this quicker and smaller? Or just skip the whole thing because the zipped versions of the repo are just so handy that they are auto generated and make releases a matter of deploying and then sending the auto generated draft release note live.

What do you two think?

Ultimately I’d like to run the deploy through GitHub Actions but still some work to do to enable that.