New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a common type for ingested tokens #70

Merged

saraswatpuneet merged 1 commit into main from ingested_tokens

Sep 21, 2023

Contributor

saraswatpuneet commented Sep 21, 2023

IngestedTokens type would contain just tokens (down the road we might have IngestedAudio etc.)
This homogenizes how all ingestors are dumping the output


          Create a common type for ingested tokens

saraswatpuneet requested a review from Ansh5461

September 21, 2023 17:05

saraswatpuneet commented

View reviewed changes

querent/common/types/ingested_tokens.py

+                      if self.file:
+                          file = str(file)
+                          self.extension = file.split(".")[-1]
+                          self.file_id = file.split("/")[-1].split(".")[0]

Contributor Author

saraswatpuneet Sep 21, 2023

basically all Ingestors returning tokens /words will have to return this type for homogenization of incoming data from ingestors

saraswatpuneet merged commit 37a9c8b into main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet