DeepBlocker is a Python package for performing blocking for entity matching using deep learning. It provides functionalities for transforming tuples into embeddings customized for blocking. Given these tuple embeddings, DeepBlocker also provides various utilities to retrieve similar tuples and construct the candidate set efficiently. DeepBlocker is self-supervised and does not require any labeled data. DeepBlocker provides multiple instantiations for tuple embedding and vector pairing for performing blocking. It is also modular and easily customizable. Each of the subcomponent is based on a pre-defined and intuitive API that allows altering and swapping up these components to achieve bespoke implementations.
For details on the architecture of the models used, take a look at our paper Deep Learning for Blocking in Entity Matching: A Design Space Exploration (VLDB '21).
All public datasets used in the paper can be downloaded from the datasets page.
We used fastText for word embedding that is pre-trained on Wikipedia.
There are four main steps in using DeepBlocker:
- Load the relevant libraries
import pandas as pd
from deep_blocker import DeepBlocker
from tuple_embedding_models import AutoEncoderTupleEmbedding, CTTTupleEmbedding, HybridTupleEmbedding
from vector_pairing_models import ExactTopKVectorPairing
import blocking_utils- Data processing: Load the relevant datasets for blocking.
left_df = pd.read_csv("left_table_csv_file_name")
right_df = pd.read_csv("right_table_csv_file_name")- Instantiate the DeepBlocker with appropriate classes for tuple embedding and vector pairing models.
tuple_embedding_model = AutoEncoderTupleEmbedding()
topK_vector_pairing_model = ExactTopKVectorPairing(K=50)
db = DeepBlocker(tuple_embedding_model, topK_vector_pairing_model)- Train the models and perform blocking of the tables. Report the accuracy.
candidate_set_df = db.block_datasets(left_df, right_df, cols_to_block)
golden_df = pd.read_csv("matches_csv_file_name")
print(blocking_utils.compute_blocking_statistics(candidate_set_df, golden_df, left_df, right_df))DeepBlocker relies on a set of external libraries that can be found in requirements.txt. You can install them as
pip install -r requirements.txt
We provide a sample script illustrating how DeepBlocker works for three major tuple embedding models -- AutoEncoder, CTT and Hybrid.
Please contact Saravanan Thirumuruganathan for any questions about the code.
DeepBlocker was developed by QCRI and University of Wisconsin-Madison. For the entire list of contributors please refer the DeepBlocker paper.