Large docs-only site: page file size becoming an issue

[gary-vertica]

Hi,

We're working on converting our 5000+ page documentation site over to Hugo+Docsy. After the conversion, I'm finding the site is not terribly responsive. It turns out that every content file is at least 1.6MB, mainly due to the entire section menu (containing all 5000 pages) repeats in every file. The total size for the converted site on disk is over 7GB. That's pretty untenable. After every page navigation, there is a pause as the browser spends time parsing/rendering the section menu.

Is there a way to get Docsy or Hugo to render just the part of the section menu that is visible when the sidebar_menu_compact parameter is set to true? There's no reason to have the rest of the hidden entries in the file.

I saw the mention of the sidebar_cache_limit setting in the documentation, but it sounds like this option only applies to how Hugo builds the site, not its output. Correct? Changing it to a lower value seems to have no effect on the file sizes.

[aimeeu]

Hi. I'm not a Docsy expert. Just wondering if you've looked at the config for the Kubernetes docs site? It has a lot of pages and is very responsive. https://github.com/kubernetes/website/blob/main/config.toml

[gary-vertica]

Yes, we looked at the Kubernetes docs, as they are one of the more prominent examples of Docsy. In their HTML output for an average page like this one the section menu takes up about 1500 lines of HTML code in a 2100-line file. So, it ends up being significant portion of the file size for each of their content pages. It's probably not enough to drag the browser down in their case, though. When you load one of our pages, the nav section freezes for a while, I assume because it takes the browser a while to crunch through a multi-level UL list that has 5000 entries.

[gary-vertica]

So, it appears other have hit a similar issue before. Issue #144 mentions a slowdown in generating large sites. There's some code that changes partials/sidebar-tree.html to limit the generation of partial menu items. Unfortunately, that partial has significantly changed since that issue was posted. I tried adapting the code to the new partial. It does sorta work... but the result is a partial menu that's truncated at the second level. I've tried my hand at changing the partial myself, but I don;t have a very good grasp of Hugo/Go templating, and my results either fail to remove unnecessary items or prevent any items below the second level of the section menu from appearing.

[gary-vertica]

OK, so, hopefully I think I've gotten this figured out.

When I implemented these changes to partials/sidebar-tree.html based on the changes from Issue #144. Specifically:

{{- if $withChild }}
  {{- $ulNr := add $ulNr 1 }}
  {{- if or ($p.IsDescendant $s) ($s.IsDescendant $p ) -}}
  <ul class="ul-{{ $ulNr }}{{ if (gt $ulNr 1)}} foldable{{end}}">
    {{ range $pages -}}
    {{ if (not (and (eq $s $p.Site.Home) (eq .Params.toc_root true))) -}}
    {{ template "section-tree-nav-section" (dict "page" $p "section" . "shouldDelayActive" $shouldDelayActive "sidebarMenuTruncate" $sidebarMenuTruncate "ulNr" $ulNr "ulShow" $ulShow )   }}
    {{- end }}
    {{- end }}
  </ul>
  {{- end }}
  {{- end }}

Noet the added {{- if or ($p.IsDescendant $s) ($s.IsDescendant $p ) -}}. When I implemented this to start with, the section menu would get truncated at the second level... you would not find anything deeper in the section menu than the second level of pages. That's unworkable for us, as we have some fairly nested content.

After adding in a bunch of debugging printf statements, I realized that the issue was the section menu was being cached and reused for all pages in each section. That caching turns out to be the problem. The section menu is only generated once for each section. So the same menu gets reused for all lower-level pages. I tried disabling this caching by setting the sidebar_cache_limit param to something much higher than the number of pages in our project. Sure enough, the section menu is now complete. The section menu is complete no matter how nested the content is.

[fekete-robert]

Hi @gary-vertica, everything I write is untested and pure speculation, but I'd try to:

1. get the toc into a separate html file, and then
2. try to include it, for example using this method: https://www.w3schools.com/w3js/w3js_html_include.asp

Step 2 is easy to test, and if that or something similar works, then the trick is to get Step 1 working. You could try to solve that problem either in Hugo, or using a postprocessing script.

• With Hugo, I'd probably create a placeholder content file that uses a custom layout that includes only the sidebar toc that you want to include. (Or maybe a custom hugo output format that does the same might be easier, I'm not sure.)
• The postprocessing method would be to write a small script (in whatever language you feel handy that can handle html structures, personally I'd go for Python and the BeautifulSoup library), that opens one of the hugo-generated html files, extracts the toc into a separate file, then opens every hugo-generated html file and replaces the toc with the reference that's needed for including the external toc file.

[fekete-robert]

Actually, after a quick test it seems to work, the only problem is that if the sidebar is set to be foldable (sidebar_menu_foldable = true
in your config file), then the sidebar menu closes every time you click on an entry. (No idea how to solve that.)
Otherwise it seems to work at a first glance, with significant savings in file size (my main page went from 99KB to 15KB).

Here is what I did:

1. Copy the themes/docsy/layouts/partials/sidebar.html file into the /layouts/partials/original-sidebar.html file (we will need that to create the toc to include into the other files)

2. Create a file called /layouts/partials/sidebar.html with the following content. This is the code that will include the generated toc file.

   <div w3-include-html="/sidebar-toc/index.html"></div>
   <script>
   w3.includeHTML();
   </script>

3. Include the w3.js file: Add the following to the head-end.html customization file of your project:

   <script src="https://www.w3schools.com/lib/w3.js"></script>

4. Create a new file /layouts/toc/baseof.html with the following content. That is the new page type that will generate only the toc into a file.

   {{ partial "original-sidebar.html" . }}

5. Create a placeholder file with the new page type: content/sidebar-toc/index.md with the following content:

   ---
   title: sidebar placeholder
   toc_hide: true
   hide_summary: true
   type: toc
   ---
   

6. Build your site. If I didn't forget anything, it should work. If anyone knows how to make the foldable version work properly, please let me know.

[gary-vertica]

That sounds like a neat workaround. especially if you solve the sidebar closing issue. I suspect there's some JavaScript in each page that resets the section menu on page load. If you can find and disable that, it should leave the menu alone.

In our solution, enabling foldable menus results in the dropdowns (the > in the section menu indicating nodes with content under them) having no effect. You click them, they rotate downwards, but nothing drops down. Clicking the page titles in the menu navigates as usual. I'll have a discussion with my team about whether being able to explore the menu without navigating to a new page is really worthwhile. I suspect it isn't.

[polarweasel]

@gary-vertica late to the party here, but is there a way you can categorize your content and then manually define smaller menus? That way, each page will only have to have a subset of the entire content tree.

We're in the middle of doing that on our much-smaller site right now. Have a look at Hugo's docs for menus.

[verticadocs]

I really like to avoid manual work on the TOC. I find automating things as much as possible limits the ability for someone to mess things up, and also reduces the annoying steps in the doc workflow.

The solution I mentioned above works OK for us. The only downside right now is that it breaks when using Hugo > 0.99. Version 0.100 changed something about how ancestors and descendants are determined in the section menu and messes up my fix. I've put off solving that until we finally deploy a version of our docs in Hugo. I noticed that Docsy 0.4 just came out and mentions some sort of fix in the section menu generation, so maybe it'll be as easy as re-implementing my change on the newer Docsy version.