skrub - formerly dirty_cat - is a Python library that facilitates prepping your tables for machine learning.
For a detailed description of the problem of encoding dirty categorical data, see Similarity encoding for learning with dirty categorical variables [1] and Encoding high-cardinality string categorical variables [2].
If you like the package, please spread the word, and ⭐ the repository!
skrub provides tools (TableVectorizer, fuzzy_join...) and
encoders (GapEncoder, MinHashEncoder...) for morphological similarities,
for which we usually identify three common cases: similarities, typos and variations
The first example notebook goes in-depth on how to identify and deal with dirty data using the skrub library.
Semantic similarities are currently not supported. For example, the similarity between car and automobile is outside the reach of the methods implemented here.
This kind of problem is tackled by Natural Language Processing methods.
skrub can still help with handling typos and variations in this kind of setting.
There are currently no PiPy releases. You can still install the package from the GitHub repository with:
pip install git+https://github.com/skrub-data/skrub.git
Dependencies and minimal versions are listed in the setup file.
Are listed on the skrub's website
If you want to encourage development of skrub, the best thing to do is to spread the word!
If you encounter an issue while using skrub, please open an issue and/or submit a pull request. Don't hesitate, you're helping to make this project better for everyone!
| [1] | Patricio Cerda, Gaël Varoquaux, Balázs Kégl. Similarity encoding for learning with dirty categorical variables. 2018. Machine Learning journal, Springer. |
| [2] | Patricio Cerda, Gaël Varoquaux. Encoding high-cardinality string categorical variables. 2020. IEEE Transactions on Knowledge & Data Engineering. |