feat: Adding Semantic Text Splitter Component (Text Splitters)

[joaoguilhermeS]

Feat: Semantic Text Splitter with Advanced Threshold Controls

Overview
Introduces a new Semantic Text Splitter component that provides flexible text chunking with statistical threshold controls and regex support.

Features:

• Multiple threshold control methods:
  • Percentile-based splitting
  • Standard deviation thresholds
  • Interquartile range
• Configurable chunk size and count
• Optional regex-based splitting

[edwinjosechittilappilly]

@joaoguilhermeS is langchain_experimental a part of the pyproject of langflow? if not @ogabrielluiz can we add it to the pyproject?
@joaoguilhermeS Can you confirm if it's already added by uv to the pyproject file?

Also, I suggest we add the beta as True in the component since it is a part of langchain_experimental

[joaoguilhermeS]

Thanks for your review @edwinjosechittilappilly I have added the langchain_experimental to the pyproject and everything seems to be working. I also added the beta flag on the Semantic Text Splitter component.

[edwinjosechittilappilly]

@joaoguilhermeS Great job.

https://github.com/langflow-ai/langflow/blob/jg/feat-semantic-text-splitter/src/backend/base/pyproject.toml
I already have langchain_experimental.

Can you test adding a different version of langchain_experimental in both?

I suggest updating the base pyproject so we won’t need to add it in other pyprojects outside of the base.

[joaoguilhermeS]

Hey @edwinjosechittilappilly, I tested the component with the current langchain-experimental dependency version and it is working just fine, so I think there is no need to update so we do not break other components. Thanks for the review.

[edwinjosechittilappilly]

LGTM.
@Cristhianzl we might need to update the uvlock accordingly later if required.

[edwinjosechittilappilly]

@joaoguilhermeS if there is no change in the dependencies then I would suggest the uvlock need not be updated.

Apart from that its all good.

[codspeed-hq]

CodSpeed Performance Report

Merging #4254 will not alter performance

_{Comparing jg/feat-semantic-text-splitter (0db0ec3) with main (fa28541)}

Summary

✅ 15 untouched benchmarks

[edwinjosechittilappilly]

LGTM

[edwinjosechittilappilly]

@ogabrielluiz I believe you might need to approve for this to get merged.(Since changes requested). Can you take a look into it, Its been blocked from merging.