tfidf-summarizer is a Rust crate for computing Term Frequency-Inverse Document Frequency (TF-IDF) vectors for a set of documents. This statistical measure is used to evaluate how important a word is to a document in a collection or corpus.
- Tokenization of text data.
- Calculation of term frequency (TF) for words in each document.
- Calculation of document frequency (DF) for words across all documents.
- Calculation of inverse document frequency (IDF) across all documents.
- Computation of TF-IDF scores for words in each document.
Add this to your Cargo.toml:
[dependencies]
tfidf-summarizer = "0.1.0"