Slow Page index with many pages

[rhartmann]

It seems that every access of the Page Index /-/index loads and parses the markdown of every .md file in the repository to assemble the ToC of each file. While I am aware this is required for Toggle page headers, I noticed somewhat slow loading times when dropping 330-ish files into Otterwiki. Header depth of most pages is 3 or below. Every hit of /-/index takes ~3-4 seconds to parse all the pages, which will cause acceptance issues with some people.

I did some tests using empty strings or loading only the first 200 bytes per file from storage. Both improved loading times of the Page Index greatly (<0.2 seconds) at the cost of not having page headers available for display. I suppose the markdown parsing is the costly part.

On top of that, showing all the page headers for 300+ pages is not very helpful. The Page Index is utterly long and cluttered when showing page headers.

I'd love to have the loading of page headers either be opt-in, cached, asynchronous (when the page already shows) or on-demand (when the user hits the toggle).

[redimp]

Hey @rhartmann, thank you for reporting this issue.

When I implemented this I thought about caching the page structure .. but I like your idea of loading this on-demand better. Will check and think about both approaches to solve this.

[rhartmann]

Maybe another useful approach: Introduce an environment variable to make it configurable. Like:

if len(md_files) > int(app.config.get('LOAD_HEADERS_PAGE_LIMIT', 50)):
  # Too many pages. Do not load page headers (and hide or disable the toggle)

This also provides opt-out (you could set it to 0 to never load page headers) and could be combined with cache, asynchronous or on-demand.

[redimp]

The code for caching the headers is ready and will be released in the next version. Working on other issues that I want to include too. Planing to release it this week. 🤞

[redimp]

Caching has been released in v2.9.0.

[rhartmann]

Loading the page index for my 330-ish pages from cache takes ~0.3 seconds (~3-4 seconds before). Serious improvement, thank you!