Create reference.zip file during build process

[runemadsen]

We need to create a reference.zip file during the build process that the Processing IDE can download when needed. This issue actually covers two main things:

• Create the reference.zip file during build
• Minimize the size of the reference.zip file

As far as minimizing the size of the reference.zip file, I have done some research to figure out next steps, and some work has already been done to fix some things. However, there are plenty of more short-term and long-term work to do.

As a quick explanation of the current bundle size: Gatsby prioritizes speed and size of loaded content over bundle size, which is great for serving over the web, but not so great if the size of the stored content matters. One example of this is that the same page content is saved in the HTML files and page-data JS files, but only one of those files will be loaded by the user depending on whether it's the initial load (HTML) or link navigation (page-data). But both are obviously contributing to the bundle size, even though some of this is de-duplicated by the zipping algorithm. On top of this comes things like responsive images, which produces 3-4 versions of each image in the final bundle, but works in the same way: Only a single image is loaded, but all of them are in the bundle. So in some way, we're going to need to fight Gatsby a bit to get the best of both worlds.

Here are some short-term things we can do:

• Remove all .map files. Easy. Little effect on bundle size
• Remove fake german translation pages. Easy. Large effect on bundle size (I need to check if this has already been done)
• Make a separate build with responsive images disabled. More work. Medium effect on bundle size.

I did a quick test of these things:

• I removed the fake German translation and .map files, and this resulted in a 60MB ZIP file. Since we don't have any translations on the site right now anyway, I think we should do this. We can then try to make the translations less heavy when we have one ready to go, but each translation is basically duplicating the site.
• I did another test where I removed some of the responsive images, and I could shave further 10MB off this. I need to look more into it, but I think it will save space.
• I tested zipping the site without images, which resulted in a 20MB file. This is not super good, but taking the thousands of generated files into account, not horrible either.

We might find more when we start working through these things, but the 50-60MB ZIP file seems very doable. We also need to localize some things, like not relying on the P5.js CDN version, but that's trivial. The longer term work will be going through the code with this constraint in mind and template by template looking at the file sizes and what optimizations can be done. This can save a lot of space based on the way Gatsby works, but I'm not sure how much and it'll take a bit more time.

[runemadsen]

@SableRaf I just updated the issue description to reflect the current state of the research for this. I can try to prioritize this in the coming weeks.

[SableRaf]

That's fantastic @runemadsen! Thanks a lot for looking into this. Pinging @benfry for the PDE side.

[runemadsen]

I've been thinking some more about this and here's what I think needs to change.

We have a kind of "chicken or the egg" problem here because we trigger the website builds by manually creating releases, so if the release needs the have the ZIP file, we need to update the release with the ZIP file once the build ends. If we keep this flow, it will look something like this:

1. Manually create release on GitHub
2. Automatically trigger a GitHub action to build the website and upload it to S3
3. The GitHub action ZIP's the website and attaches the ZIP to the release

The result of this is that any release will have a 10-15 minute delay before it has a ZIP file attached.

Instead, I'm thinking that it would be preferred if we flipped the logic around, so we start manually triggering the GitHub action via the UI which then will build and upload the website and then create a release with a ZIP file attached. This flow will look something like this:

1. Manually run the action on GitHub
2. Build the website and upload it to S3
3. Zip website and create a new release with the ZIP file

@SableRaf @benfry What do you think about this? If you think it's a good approach, we can get going on it.

[SableRaf]

Hi @runemadsen thanks for the update. So just to make sure I understand: in your suggested approach, we wouldn't have to manually create a new release but instead just run a Github action that builds/uploads the website, then zips it and creates the release?

[runemadsen]

Yes, it will basically be the same thing, but you will have to press "Run Action" instead of creating a release. We could use the package.json version property to automatically fill out the release number (which is not used for anything right now, I think)

[SableRaf]

Yeah the release number isn't used for anything meaningful. Would there be a way to add a description for the release when needed?

[runemadsen]

I was just looking at this, and we can use input parameters for the GitHub action to allow you to specify a version and description when running the action in the GitHub interface.

[runemadsen]

Update on this: I pretty much have everything ready to go, and I just need to take some time to test it on the live repo. The problem is that manually triggered actions only work on main, so I'm going to need to disable the existing action and add a new action to main while testing this out. In the process, I will be making some new releases that hopefully have zip files attached to them.

@SableRaf Is this okay with you?

[SableRaf]

Yes that sounds good! Thanks for asking @runemadsen. I just made a release to get the latest changes out of the way so you're all clear for testing the new GitHub action. Make sure to label those releases as tests.

[runemadsen]

Great, will do!

[runemadsen]

@SableRaf @benfry This has been implemented! You'll see that I made a test release that has the reference.zip file attached to the release. This happens automatically, and I like this deployment method much better. I also updated the README to explain how to deploy the website with the new format.

I vote for closing this issue and opening a new one linked to this one named "Minimize size of reference.zip file" so we can track the needs in a separate issue.

[SableRaf]

Great! That's a huge improvement. Thanks @runemadsen ✨

I vote for closing this issue and opening a new one linked to this one named "Minimize size of reference.zip file" so we can track the needs in a separate issue.

That sounds good. Do you want to go ahead and create the new issue or should I open one?

[runemadsen]

Great! I can create the issue now.