rustdoc produces gigantic on-disk content in modules with large number of numerical constants

[elinorbgr]

We have in Smithay some modules that define a large number of numerical constants (const u32), and this triggers a pathological edge-case in terms of documentation generation: rustdoc generates one 4.6 KB file for each of these constants.

For example see this module: https://smithay.github.io/smithay/smithay/wayland/seat/keysyms/index.html

It contains something like ~2400 numerical constants, as a result the documentation folder for this module alone is 20 MB. We have a few other instances of this in Smithay, and we realized that because of this our gh-pages branches is more than 99% of the size of the github repo (the total size of the target/doc folder for smithay and its public dependencies is around 220MB), and it significantly impacts clone and pull times for our contributors.

We're going to take steps to mitigate this on our side, but I suppose maybe something can be done on rustdoc side as well?

cc @GuillaumeGomez who asked me to make this issue.

edit:

Giving more details on this, more generally the issues lies with modules that contain a large number of un-documented items (like functions or constants), like the ones generated by gl_generator for example.

For such modules, rustdoc will create one file per item, with a constant 4.6KB overhead per file, while as a documentation reader this really does not bring anything useful given those pages do not contain any additional documentation beyond the symbol prototype.

[GuillaumeGomez]

So I think there are a few leads on how to address this:

• Group all items like constants or functions into a same file (one for constants, one for functions, etc). The problem with this solution is that it's not backward compatible.
• Remove items from the same module in the sidebar.
• Try to shorten the DOM even more, but I don't think the impact would be big.
• Others?

[notriddle]

Remove items from the same module in the sidebar.

I'm not sure tweaking the sidebar would accomplish much. The keysyms/sidebar-items.js is a 64K list of names, while keysyms/index.html is 620K of tags.

[elinorbgr]

Providing some update about this part:

we realized that because of this our gh-pages branches is more than 99% of the size of the github repo (the total size of the target/doc folder for smithay and its public dependencies is around 220MB), and it significantly impacts clone and pull times for our contributors.

Actually, updating our CI script to not store the history on gh-pages and instead just overwrite the branch every time drastically improved the situation. While the total size of the generated documentation is still 220 MB, it appears that git is able to compress it very well, as now a fresh pull of the smithay repository is only 15 MB (as opposed to ~450 MB when we still had gh-pages history).