Add step to deduplicate records based on embeddings

[plaguss]

Description

This PR adds an EmbeddingDedup step to deduplicate texts based on embeddings, and previously detected nearest neigbours. The typical workflow is the following:

with Pipeline() as pipeline:
    loader = LoadDataFromDicts(
        data=...,  # Records containing the texts and embeddings
    )

    # NOTE: Guide to choose an index: https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index
    nn = FaissNearestNeighbour(
        k=3,
        metric_type=faiss.METRIC_INNER_PRODUCT,
        search_batch_size=50,
        # This fields would be needed in case of training an index (a big dataset)
        # string_factory="IVF300_HNSW32,Flat",
        # train_size=len(dataset),
    )

    embedding_dedup = EmbeddingDedup(
        threshold=0.99,
    )
    loader >> nn >> embedding_dedup

distiset = pipeline.run(use_cache=False)

ds = distiset["default"]["train"]
ds_dedup = ds.filter(lambda x: x["keep_row_after_embedding_filtering"])

[github-actions]

Documentation for this PR has been built. You can view it at: https://distilabel.argilla.io/pr-946/

[codspeed-hq]

CodSpeed Performance Report

Merging #946 will not alter performance

_{Comparing dedup-embeddings (3e317af) with develop (973e0fa)}

Summary

✅ 1 untouched benchmarks

[gabrielmbmb]

LGTM! Much needed step :)