Improve split and merge #1612

pseudotensor · 2024-05-11T08:46:44Z

do semantic splitting for input chunking and summary-extraction chunking if possible
Interpret chunk_size as tokens if can
do sentence chunking if can
fix loss of text if original input is large like for summary, where it was chopping off (after split) all but 1 chunk
fix counting of tokens >= vs. >
fix doc show for large text, file type was full text
improve text doc view so wraps as sentences

…ntence splitting, and multilingual

pseudotensor added 9 commits May 10, 2024 22:28

Fix split and merge

67c8a57

Generalize split_merge_docs to handle semantic splitting (if GPU), se…

e65cd15

…ntence splitting, and multilingual

Update test

9aa4ab1

Semantic or sentence splitting for normal chunking too

3406e93

Break pure text in doc view by sentences

8d2af1d

Fix top_k_docs filtering when had one large block at first

ad975db

Fix input and output tokens counting for summarization

fbf4143

Update test

d86538b

Fix exception

819ea41

pseudotensor marked this pull request as ready for review May 11, 2024 08:48

pseudotensor merged commit ae586c6 into main May 11, 2024
2 checks passed

pseudotensor deleted the fix_split_merge branch May 11, 2024 08:49

Provide feedback