Repository size & contribution issues

[hinell]

Currently the repository is well over ~500MB in size and fetching down this volume is time consuming process that prevents people from contributing quickly & easily even if cloned by git clone --depth ...

It also increases a checkout-time. I propose maintainers to adopt a practice of keeping only tagged revisions of the project and resort to git merge --squashing & git rebase-ing in order to make contribution process reasonable.

UPD: When I say tagged (currently tags aren't used), I mean that major versions are kept and outdated ones (and history inbetween them) are dropped altogether!

Thanks!

[teoli2003]

We already do git merge --squash on this repository.

[Josh-Cena]

I'm also concerned about the size of the repo and the speed at which it's growing. At the same time, I find the version history quite useful as I like to blame some lines that caused me to raise eyebrows and check the rationale for that change. I don't know if we can get the best of both worlds, though. I think a good way to proceed is to encourage contributors to merge typo fixes into batches, instead of editing one file per PR — I understand that makes the bar much higher, but we have to educate them.

even if cloned by git clone --depth ...

That's interesting. Why is --depth 1 taking long?

[nschonni]

It's not as bad as https://github.com/MicrosoftDocs/azure-docs/, but the amount of unique strings constantly changing is always going to be an issue for git.

[hinell]

That's interesting. Why is --depth 1 taking long?

@Josh-Cena It doesn't. Contributions from a shallow repo are prevented by a shallow update not allowed error.
To avoid this error one has to download entire repository.

[nschonni]

A little related, but the meta folder/site also is using a git submodule. If that site is not being used, that submodule could be removed

[hinell]

As of April 19, 2023 when docs are built, the size of the built/ folder is over 1GB. The repo takes 501MB total. 🤯

[teoli2003]

75% of the built/ folder are HTML files (and images) (and the 25% remaining json & metadata). Given the size of MDN, this doesn't shock me (We can have several built folders on one entry-level USB key, < 10$ – I am not saying it is a good idea to build it on a USB key, just to get an idea of the price). I don't think this is a problem as this directory is not stored in git. You don't need to build it, ultimately.

For the repo size, it would be interesting to have a ratio of commits made locally vs using the UI per country to assess the extent of the problem.

[hinell]

@teoli2003 Well pulling down such bit repo actually can prevent potential contributors from contributing. You can't contribute from a shallow repo.

[Josh-Cena]

You can't contribute from a shallow repo

Sure you can! The DT repo is probably one or two orders of magnitude larger than us so they have much better documentation on this: https://github.com/DefinitelyTyped/DefinitelyTyped/blob/master/README.md#partial-clone

I'm not sure what prevents PRs—the few times I used it, shallow clone on itself worked relatively well (although sparse checkout was a beast to tame). You follow the same branch/commit/push process.

[hinell]

@Josh-Cena Oh yeah, I forgot about sparse checkout. Really nice thing.