docs: add release notes and docs for jusText OOM fix (PR #1534) - #1697
Conversation
Greptile SummaryThis PR adds release notes and documentation for the Confidence Score: 5/5Safe to merge — documentation-only changes with no runtime code affected All prior P1 concerns from previous review threads (duplicate ## Bug Fixes heading, orphaned Audio Stage Name Propagation subsection) are resolved. The new content is accurate and internally consistent: the No files require special attention for this PR Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["CommonCrawlDownloadExtractStage\nextractor_max_calls_per_worker=2 (default for JusText)"] --> B["Ray Data Worker starts"]
B --> D["Process WARC task\n(jusText/lxml extraction)"]
D --> C{"task_count >=\nmax_calls_per_worker?"}
C -- No --> D
C -- Yes --> E["Recycle worker process\n(reclaim fragmented C-heap)"]
E --> B
Reviews (5): Last reviewed commit: "docs: add release notes and docs for jus..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| ## Bug Fixes | ||
|
|
||
| ### JusText Extraction OOM (PR #1534) | ||
|
|
||
| Fixed out-of-memory errors during long-running jusText extraction jobs. Memory fragmentation from lxml/libxml2 C-heap allocations caused RSS growth across workers until OOM. Worker recycling via `max_calls_per_worker` now prevents this by restarting worker processes periodically. | ||
|
|
There was a problem hiding this comment.
## Bug Fixes section
The new JusText OOM entry introduces a second ## Bug Fixes heading at line 31, but an existing ## Bug Fixes section already exists further down the file (after the ## Dependency Updates block, at line 42). This produces two top-level sections with the same title, which breaks document navigation (duplicate anchor IDs) and fragments the bug-fix list for readers.
The JusText OOM entry should be moved inside the existing ## Bug Fixes section so both fixes live together. Remove this standalone ## Bug Fixes heading and its content block here, and instead add a ### JusText Extraction OOM (PR #1534) subsection inside the existing ## Bug Fixes section (currently at line 42), alongside ### Audio Stage Name Propagation (PR #1470).
769fc94 to
93b5972
Compare
| ## Dependency Updates | ||
|
|
||
| - **Cosmos-Xenna**: Updated from 0.1.2 to 0.2.0 with simplified resource model | ||
| - **Ray**: Updated to 2.54 | ||
|
|
||
| ## Bug Fixes | ||
|
|
||
| ### Audio Stage Name Propagation (PR #1470) | ||
|
|
||
| Fixed audio pipeline stage names not propagating in `StagePerfStats`, making benchmark output unable to identify per-stage timing. All audio stages (`GetAudioDurationStage`, `PreserveByValueStage`, `AudioToDocumentStage`, `GetPairwiseWerStage`) now correctly report their names, and stage performance history persists when stages create new task objects. |
There was a problem hiding this comment.
Audio Stage Name Propagation is orphaned under Dependency Updates
When fixing the duplicate ## Bug Fixes heading, the original ## Bug Fixes heading that preceded ### Audio Stage Name Propagation was removed but the subsection itself was kept in place. As a result, ### Audio Stage Name Propagation (PR #1470) now sits directly inside the ## Dependency Updates section with no parent bug-fixes heading — it reads as a dependency update, not a bug fix.
The correct fix is to consolidate both bug-fix entries under a single ## Bug Fixes section. One clean approach is to restore the ## Bug Fixes heading after ## Dependency Updates and place both subsections inside it:
| ## Dependency Updates | |
| - **Cosmos-Xenna**: Updated from 0.1.2 to 0.2.0 with simplified resource model | |
| - **Ray**: Updated to 2.54 | |
| ## Bug Fixes | |
| ### Audio Stage Name Propagation (PR #1470) | |
| Fixed audio pipeline stage names not propagating in `StagePerfStats`, making benchmark output unable to identify per-stage timing. All audio stages (`GetAudioDurationStage`, `PreserveByValueStage`, `AudioToDocumentStage`, `GetPairwiseWerStage`) now correctly report their names, and stage performance history persists when stages create new task objects. | |
| ## Dependency Updates | |
| - **Cosmos-Xenna**: Updated from 0.1.2 to 0.2.0 with simplified resource model | |
| - **Ray**: Updated to 2.54 | |
| ## Bug Fixes | |
| ### JusText Extraction OOM (PR #1534) | |
| Fixed out-of-memory errors during long-running jusText extraction jobs. Memory fragmentation from lxml/libxml2 C-heap allocations caused RSS growth across workers until the process ran out of memory. Worker recycling through `max_calls_per_worker` now prevents this by restarting worker processes periodically. | |
| ### Audio Stage Name Propagation (PR #1470) |
This removes the standalone ## Bug Fixes block introduced above ## Dependency Updates and reunites both bug-fix entries under one correctly placed section.
| - **Automatic Worker Recycling for JusText**: `CommonCrawlDownloadExtractStage` now defaults to `extractor_max_calls_per_worker=2` when using `JusTextExtractor`, automatically restarting worker processes every two tasks to reclaim fragmented memory. | ||
| - **Configurable `max_calls_per_worker`**: `DocumentIterateExtractStage` accepts a new `max_calls_per_worker` parameter that instructs the Ray Data executor to restart worker processes after a specified number of tasks. This is useful for any extraction stage that uses C libraries prone to heap fragmentation, such as lxml. | ||
| - **`extractor_max_calls_per_worker` on Composite Stages**: `DocumentDownloadExtractStage` and `CommonCrawlDownloadExtractStage` expose this setting through `extractor_max_calls_per_worker`, which propagates to the underlying iterate-extract stage. |
There was a problem hiding this comment.
Seems pretty verbose. I'm okay with having smaller 1-2 line sections here unless more descriptive options and preferred that'll be summarized later on
|
|
||
| ### JusText Extraction OOM (PR #1534) | ||
|
|
||
| Fixed out-of-memory errors during long-running jusText extraction jobs. Memory fragmentation from lxml/libxml2 C-heap allocations caused RSS growth across workers until the process ran out of memory. Worker recycling through `max_calls_per_worker` now prevents this by restarting worker processes periodically. |
There was a problem hiding this comment.
@lbliii Do you have a preference on keeping it in both places or just one?
Addresses PR #1697 review feedback: - Condensed "What's New" entry to 1-2 lines (ayushdg verbosity feedback) - Consolidated all bug fixes under single "## Bug Fixes" section - Fixed "Audio Stage Name Propagation" being orphaned under "Dependency Updates" - Kept feature in "What's New" and bug fix in "Bug Fixes" as separate concise entries Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Lawrence Lane <llane@nvidia.com>
2 participants
Description
Adds documentation and release notes for PR #1534, which introduces
max_calls_per_workersupport to mitigate OOM errors caused by lxml/libxml2 memory fragmentation during long-running jusText extraction jobs. Release notes cover the new worker recycling feature and the OOM bug fix. The Common Crawl guide adds theextractor_max_calls_per_workerparameter and a new "Memory Management for Extraction" section with usage examples. The memory management guide cross-links to the new section.Checklist