[PERF]: pipeline segment committing/flushing

[codetheweb]

Description of changes

This introduces two new operators that allow committing and flushing individual segments. This enables pipelining across all segment types.

Some changes in service of this:

• write_to_blockfiles() is renamed to a generic finish() method on the SegmentWriter trait.
• There is an enum over all segment writer and flusher types to allow for enum dispatch.
• Flushers are now clonable because .flush() consumes self but operators receive &input. I don't love this, I think the alternative is allowing operators to consume the input.
• Tasks for each segment are grouped together under the same span.

An example trace (with two partitions):

Screen.Recording.2024-11-20.at.5.09.37.PM.mov

Test plan

How are these changes tested?

• Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the docs repository?

n/a

[github-actions]

Please tag your PR title with one of: [ENH | BUG | DOC | TST | BLD | PERF | TYP | CLN | CHORE]. See https://docs.trychroma.com/contributing#contributing-code-and-ideas

[github-actions]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• Can you think of any use case in which the code does not behave as intended? Have they been tested?
• Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• If appropriate, are there adequate property based tests?
• If appropriate, are there adequate unit tests?
• Should any logging, debugging, tracing information be added or removed?
• Are error messages user-friendly?
• Have all documentation changes needed been made?
• Have all non-obvious changes been commented?

System Compatibility

• Are there any potential impacts on other parts of the system or backward compatibility?
• Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[codetheweb]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• [PERF]: pipeline segment committing/flushing #3163 👈 (View in Graphite)
• [PERF]: parallelize applying log to segment types #3134
• [ENH]: move materialization into operator #3166
• [ENH]: make log materialization result owned #3227
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[codetheweb]

Superseded by #3360. Creating a new branch was easier than rebasing as the approach changed slightly.