[Feature] New transform to remove repeating text sequences from documents

[Harmedox]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

Feature

In the current dedupe process, duplicate documents are identified. However, in certain situations, boilerplates such as menu items appear in the extracted text, repeating over multiple documents. In this case, we only care about removing the repeated sequence of substrings.

What I propose is:

• a new transform - e.g., sequence-deduplication
• identify and remove all substrings of a given length (length_threshold) that are repeated more times than a set number (frequency_threshold)
• option to retain the first copy of the duplicated sequence
• modify the content in-place and not annotate - for efficient memory usage

Are you willing to submit a PR?

• Yes I am willing to submit a PR!