Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Add text embedder #996

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
98ed529
Sentence Embedder API using sentence transformers
abhijithneilabraham Nov 24, 2021
83fbf1e
remove train, test and pred step
abhijithneilabraham Nov 24, 2021
4ad3abb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Nov 24, 2021
8cd9395
sentence embedders with forward step and predict step
abhijithneilabraham Dec 5, 2021
47bbce7
Merge branch 'master' into ST_embeddings
abhijithneilabraham Dec 5, 2021
34f39d1
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 5, 2021
219042a
Update __init__.py
abhijithneilabraham Dec 5, 2021
9f9a965
Merge branch 'ST_embeddings' of https://github.com/abhijithneilabraha…
abhijithneilabraham Dec 5, 2021
4b3c772
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 5, 2021
5fecb22
Merge branch 'master' into ST_embeddings
ethanwharris Dec 8, 2021
06e35bd
Updates
ethanwharris Dec 8, 2021
2071c9f
Create test_model.py
abhijithneilabraham Dec 8, 2021
5477415
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 8, 2021
8db110d
__init__ for embedding
abhijithneilabraham Dec 8, 2021
2c09c71
Merge branch 'master' into ST_embeddings
abhijithneilabraham Dec 8, 2021
a6bfc9f
remove download_data()
abhijithneilabraham Dec 8, 2021
602d9e1
Merge branch 'ST_embeddings' of https://github.com/abhijithneilabraha…
abhijithneilabraham Dec 8, 2021
b29b6e0
Merge branch 'master' into ST_embeddings
abhijithneilabraham Dec 9, 2021
e771883
Merge branch 'master' into ST_embeddings
abhijithneilabraham Dec 9, 2021
21305d6
lower size model for text embededer examples and test
abhijithneilabraham Dec 9, 2021
5d1b4c6
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 9, 2021
9570522
text embedder example entry to CI
abhijithneilabraham Dec 9, 2021
8cfd877
Merge branch 'master' into ST_embeddings
abhijithneilabraham Dec 9, 2021
bb98d77
change `SentenceEmbedder` to `TextEmbedder`
abhijithneilabraham Dec 9, 2021
923e6ec
remove `download_data` import
abhijithneilabraham Dec 9, 2021
8c90286
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 9, 2021
20233f2
fix bug - test_model.py
abhijithneilabraham Dec 9, 2021
33f30e6
Merge branch 'ST_embeddings' of https://github.com/abhijithneilabraha…
abhijithneilabraham Dec 9, 2021
57aa577
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 9, 2021
fdbb2de
Update test_model.py
abhijithneilabraham Dec 9, 2021
6f5b9e8
Merge branch 'ST_embeddings' of https://github.com/abhijithneilabraha…
abhijithneilabraham Dec 9, 2021
14a5e27
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Dec 9, 2021
3d14659
Update CHANGELOG.md
abhijithneilabraham Dec 9, 2021
f69f207
Merge branch 'master' into ST_embeddings
tchaton Dec 9, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion flash_examples/text_embedder.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,10 @@
)

# 2. Load a previously trained SentenceEmbedder
model = SentenceEmbedder(backbone="sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
model = SentenceEmbedder(backbone="sentence-transformers/all-MiniLM-L6-v2")
ethanwharris marked this conversation as resolved.
Show resolved Hide resolved

# 3. Generate embeddings for the first 3 graphs
trainer = flash.Trainer(gpus=torch.cuda.device_count())
predictions = trainer.predict(model, datamodule=datamodule)
print(predictions)

2 changes: 1 addition & 1 deletion tests/text/embedding/test_model.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@

# ==============================

TEST_BACKBONE = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2" # super small model for testing
TEST_BACKBONE = "sentence-transformers/all-MiniLM-L6-v2" # super small model for testing
model = SentenceEmbedder(backbone=TEST_BACKBONE)


Expand Down