Skip to content

Autogenerate preset table #690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mattdangerw merged 14 commits into from
Feb 22, 2023
Merged

Autogenerate preset table #690

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
a9ab43c
Autogenerate preset table
Cyber-Machine Jan 24, 2023
0709f10
Made suggested changes
Cyber-Machine Jan 25, 2023
12d269b
Removed prev desc and formatted file.
Cyber-Machine Jan 26, 2023
197fa01
Merge branch 'keras-team:master' into preset_table
Cyber-Machine Jan 26, 2023
6b36cb0
Added official_name and paths to metadata
Cyber-Machine Feb 4, 2023
dfae988
Merge branch 'keras-team:master' into preset_table
Cyber-Machine Feb 4, 2023
75eb17a
Merge branch 'preset_table' of https://github.com/Cyber-Machine/keras…
Cyber-Machine Feb 4, 2023
3f3a12c
Formatted the code using black
Cyber-Machine Feb 4, 2023
056ecf2
Updated metadata for ALBERT, BART, DeBERTa, F_Net
Cyber-Machine Feb 21, 2023
58be680
Merge branch 'keras-team:master' into preset_table
Cyber-Machine Feb 21, 2023
8a43bb6
Minor content updates
mattdangerw Feb 22, 2023
1d4794a
Merge branch 'master' into preset_table
mattdangerw Feb 22, 2023
2986c7b
Format fixes
mattdangerw Feb 22, 2023
a70291c
One last round of updates
mattdangerw Feb 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 1 addition & 5 deletions keras_nlp/layers/masked_lm_mask_generator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,11 +147,7 @@ def call(self, inputs):
# convert dense to ragged.
inputs = tf.RaggedTensor.from_tensor(inputs)

(
token_ids,
mask_positions,
mask_ids,
) = tf_text.mask_language_model(
(token_ids, mask_positions, mask_ids,) = tf_text.mask_language_model(
Copy link
Member

@mattdangerw mattdangerw Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you update your install of black (pip install -U black), you should get the latest formatting here.

inputs,
item_selector=self._random_selector,
mask_values_chooser=self._mask_values_chooser,
Expand Down
44 changes: 28 additions & 16 deletions keras_nlp/models/albert/albert_presets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,13 @@
"num_segments": 2,
},
"preprocessor_config": {},
"description": (
"Base size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"Base size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 11683584,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/albert_base_en_uncased/v1/model.h5",
"weights_hash": "b83ccf3418dd84adc569324183176813",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/albert_base_en_uncased/v1/vocab.spm",
Expand All @@ -54,10 +57,13 @@
"num_segments": 2,
},
"preprocessor_config": {},
"description": (
"Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 17683968,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/albert_large_en_uncased/v1/model.h5",
"weights_hash": "c7754804efb245f06dd6e7ced32e082c",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/albert_large_en_uncased/v1/vocab.spm",
Expand All @@ -78,10 +84,13 @@
"num_segments": 2,
},
"preprocessor_config": {},
"description": (
"Extra Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"Extra Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 58724864,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/albert_extra_large_en_uncased/v1/model.h5",
"weights_hash": "713209be8aadfa614fd79f18c9aeb16d",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/albert_extra_large_en_uncased/v1/vocab.spm",
Expand All @@ -102,10 +111,13 @@
"num_segments": 2,
},
"preprocessor_config": {},
"description": (
"Extra Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"Extra Large size of ALBERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 222595584,
Copy link
Member

@mattdangerw mattdangerw Feb 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can actually add "official_name" and "path" for all models, even ones that are not yet documented on keras.io.

The path should always match the directory the model is in in KerasNLP. The official_name should match the original paper.

},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/albert_extra_extra_large_en_uncased/v1/model.h5",
"weights_hash": "a835177b692fb6a82139f94c66db2f22",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/albert_extra_extra_large_en_uncased/v1/vocab.spm",
Expand Down
121 changes: 85 additions & 36 deletions keras_nlp/models/bert/bert_presets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,10 +30,15 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"Tiny size of BERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"2-layer BERT model where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 4386178,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_en_uncased/v1/model.h5",
"weights_hash": "c2b29fcbf8f814a0812e4ab89ef5c068",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_en_uncased/v1/vocab.txt",
Expand All @@ -53,10 +58,15 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"Small size of BERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"4-layer BERT model where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 28764674,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_small_en_uncased/v1/model.h5",
"weights_hash": "08632c9479b034f342ba2c2b7afba5f7",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_small_en_uncased/v1/vocab.txt",
Expand All @@ -76,10 +86,15 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"Medium size of BERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"8-layer BERT model where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 41374210,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_medium_en_uncased/v1/model.h5",
"weights_hash": "bb990e1184ec6b6185450c73833cd661",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_medium_en_uncased/v1/vocab.txt",
Expand All @@ -99,10 +114,15 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"Base size of BERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"12-layer BERT model where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 109483778,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_en_uncased/v1/model.h5",
"weights_hash": "9b2b2139f221988759ac9cdd17050b31",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_en_uncased/v1/vocab.txt",
Expand All @@ -122,10 +142,15 @@
"preprocessor_config": {
"lowercase": False,
},
"description": (
"Base size of BERT where case is maintained. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"12-layer BERT model where case is maintained. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 109483778,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_en/v1/model.h5",
"weights_hash": "f94a6cb012e18f4fb8ec92abb91864e9",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_en/v1/vocab.txt",
Expand All @@ -145,7 +170,14 @@
"preprocessor_config": {
"lowercase": False,
},
"description": ("Base size of BERT. Trained on Chinese Wikipedia."),
"metadata": {
"description": (
"12-layer BERT model. Trained on Chinese Wikipedia."
),
"params": 102269186,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_zh/v1/model.h5",
"weights_hash": "79afa421e386076e62ab42dad555ab0c",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_zh/v1/vocab.txt",
Expand All @@ -165,10 +197,14 @@
"preprocessor_config": {
"lowercase": False,
},
"description": (
"Base size of BERT. Trained on trained on Wikipedias of 104 "
"languages."
),
"metadata": {
"description": (
"12-layer BERT model where case is maintained. Trained on trained on Wikipedias of 104 languages"
),
"params": 177854978,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_multi/v1/model.h5",
"weights_hash": "b0631cec0a1f2513c6cfd75ba29c33aa",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_multi/v1/vocab.txt",
Expand All @@ -188,10 +224,13 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"Large size of BERT where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"24-layer BERT model where all input is lowercased. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 335143938,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_en_uncased/v1/model.h5",
"weights_hash": "cc5cacc9565ef400ee4376105f40ddae",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_en_uncased/v1/vocab.txt",
Expand All @@ -211,10 +250,15 @@
"preprocessor_config": {
"lowercase": False,
},
"description": (
"Base size of BERT where case is maintained. "
"Trained on English Wikipedia + BooksCorpus."
),
"metadata": {
"description": (
"24-layer BERT model where case is maintained. "
"Trained on English Wikipedia + BooksCorpus."
),
"params": 333581314,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_en/v1/model.h5",
"weights_hash": "8b8ab82290bbf4f8db87d4f100648890",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_en/v1/vocab.txt",
Expand Down Expand Up @@ -244,9 +288,14 @@
"preprocessor_config": {
"lowercase": True,
},
"description": (
"bert_tiny_en_uncased backbone fine-tuned on the glue/sst2 dataset."
),
"metadata": {
"description": (
"The bert_tiny_en_uncased backbone model fine-tuned on the SST-2 sentiment analysis dataset."
),
"params": 4386178,
"official_name": "BERT",
"path": "bert",
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_en_uncased_sst2/v1/model.h5",
"weights_hash": "1f9c2d59f9e229e08f3fbd44239cfb0b",
"vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_en_uncased_sst2/v1/vocab.txt",
Expand Down
55 changes: 35 additions & 20 deletions keras_nlp/models/deberta_v3/deberta_v3_presets.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,13 @@
"bucket_size": 256,
},
"preprocessor_config": {},
"description": (
"Extra small size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"metadata": {
"description": (
"Extra small size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"params": 70830722,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_extra_small_en/v1/model.h5",
"weights_hash": "d8e10327107e5c5e20b45548a5028619",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_extra_small_en/v1/vocab.spm",
Expand All @@ -47,10 +50,13 @@
"bucket_size": 256,
},
"preprocessor_config": {},
"description": (
"Small size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"metadata": {
"description": (
"Small size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"params": 141896450,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_small_en/v1/model.h5",
"weights_hash": "84118eb7c5a735f2061ecccaf71bb888",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_small_en/v1/vocab.spm",
Expand All @@ -68,10 +74,13 @@
"bucket_size": 256,
},
"preprocessor_config": {},
"description": (
"Base size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"metadata": {
"description": (
"Base size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"params": 184423682,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_base_en/v1/model.h5",
"weights_hash": "cebce044aeed36aec9b94e3b8a255430",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_base_en/v1/vocab.spm",
Expand All @@ -89,10 +98,13 @@
"bucket_size": 256,
},
"preprocessor_config": {},
"description": (
"Large size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"metadata": {
"description": (
"Large size of DeBERTaV3. "
"Trained on English Wikipedia, BookCorpus and OpenWebText."
),
"params": 435063810,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_large_en/v1/model.h5",
"weights_hash": "bce7690f358a9e39304f8c0ebc71a745",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_large_en/v1/vocab.spm",
Expand All @@ -110,10 +122,13 @@
"bucket_size": 256,
},
"preprocessor_config": {},
"description": (
"Base size of DeBERTaV3. "
"Trained on the 2.5TB multilingual CC100 dataset."
),
"metadata": {
"description": (
"Base size of DeBERTaV3. "
"Trained on the 2.5TB multilingual CC100 dataset."
),
"params": 278810882,
},
"weights_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_base_multi/v1/model.h5",
"weights_hash": "26e5a824b26afd2ee336835bd337bbeb",
"spm_proto_url": "https://storage.googleapis.com/keras-nlp/models/deberta_v3_base_multi/v1/vocab.spm",
Expand Down
Loading