Skip to content

Commit

Permalink
Add a hint for indicating that a Tree is topologically sorted (#230)
Browse files Browse the repository at this point in the history
* Regenerate the Go source code for the Remote Execution protocol

* Add a hint for indicating that a Tree is topologically sorted

I'm currently trying to improve the performance of handling of large
output directories (Tree messages), having sizes in the order of
hundreds of megabytes. In the process, I have realised that there is a
lot of value in enforcing that the Directory messages contained in them
are topologically sorted. Two practical use cases:

- When instantiating the contents of a Tree on a local file system,
  having the Tree be topologically sorted allows you to immediately
  create files and directories in the right place.

- When needing to resolve the properties of a single file by path, a
  topologically sorted Tree permits resolution by doing a simple forward
  scan.

Especially when other features like compression are taken into account,
it's useful if Tree messages can be processed in a streaming manner.

One practical issue is that most Protobuf libraries don't offer APIs for
processing messages in a streaming manner. This means that implementors
who want to achive these optimisations will need to write their own
message parsers; at least for the Tree tree itself. To make this as
painless as possible, we also require that the Tree is stored in some
normal form.

Fixes: #229
  • Loading branch information
EdSchouten authored Oct 21, 2022
1 parent aa29b91 commit f608918
Show file tree
Hide file tree
Showing 2 changed files with 606 additions and 518 deletions.
Loading

0 comments on commit f608918

Please sign in to comment.