[Feature] New transform to collect token stats

[Harmedox]

Search before asking

• I searched the issues and found no similar issues.

Component

Transforms/Other

Feature

The goal of this transform is to provide token-specific metrics annotation that could serve as an indicator of document quality.

Consideration:

• in addition to the input parquet files, access must be provided to the folder containing the tokenized (arrow) files.
• for each input parquet file, a matching arrow file must be read. If no matching arrow file is found, such a file should be skipped

For each document, annotate with the:

• doc_num_tokens - number of tokens for the document
• doc_size_kbs - document size in kb
• doc_num_chars - number of characters in the document
• tokens_per_doc_size - ratio of the number of tokens and document size
• tokens_per_doc_num_chars - ratio of the number of tokens and the number of characters in the document

Are you willing to submit a PR?

• Yes I am willing to submit a PR!