Skip to content

[DocTest] Doctest for Electra Pytorch #16628

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
bhadreshpsavani wants to merge 27 commits into from
Closed

[DocTest] Doctest for Electra Pytorch #16628

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
b34eb19
Doctest for Pytorch
bhadreshpsavani Apr 6, 2022
509ca47
Merge branch 'main' into add-doctests-electra
bhadreshpsavani Apr 6, 2022
ce8716b
upaded model file
bhadreshpsavani Apr 6, 2022
c475e5e
fixes additional space in model name
bhadreshpsavani Apr 6, 2022
bd3ab18
Empty commit
bhadreshpsavani Apr 7, 2022
e6352b2
Merge branch 'huggingface:main' into add-doctests-electra
bhadreshpsavani Apr 7, 2022
cca3a14
Empty commit
bhadreshpsavani Apr 7, 2022
dc99180
Fix doc example (#16448)
NielsRogge Apr 7, 2022
09a272b
Add inputs vector to calculate metric method (#16461)
lmvasque Apr 7, 2022
080e42d
[megatron-bert-uncased-345m] fix conversion (#16639)
stas00 Apr 7, 2022
389f661
Remove parent/child tests in auto model tests (#16653)
sgugger Apr 7, 2022
4099817
Updated _load_pretrained_model_low_mem to check if keys are in the st…
FrancescoSaverioZuppichini Apr 7, 2022
3e26e78
Update Support image on README.md (#16615)
BritneyMuller Apr 7, 2022
af14c61
RegNet (#16188)
FrancescoSaverioZuppichini Apr 7, 2022
33cb211
bert: properly mention deprecation of TF2 conversion script (#16171)
stefan-it Apr 7, 2022
4ef0abb
Add TAPEX (#16473)
NielsRogge Apr 8, 2022
9db2eeb
add vit tf doctest with @add_code_sample_docstrings (#16636)
johko Apr 8, 2022
5db2fcc
Fix error in doc of `DataCollatorWithPadding` (#16662)
secsilm Apr 8, 2022
9a24b97
Fix style
sgugger Apr 8, 2022
ab22966
Fix QA sample (#16648)
ydshieh Apr 8, 2022
d57da99
Add tests for no_trainer and fix existing examples (#16656)
muellerzr Apr 8, 2022
50f7da3
Doctest for Pytorch
bhadreshpsavani Apr 6, 2022
de66e69
upaded model file
bhadreshpsavani Apr 6, 2022
de5638f
fixes additional space in model name
bhadreshpsavani Apr 6, 2022
cadbb8c
Empty commit
bhadreshpsavani Apr 7, 2022
d1ce7dd
Empty commit
bhadreshpsavani Apr 7, 2022
454f586
Merge branch 'add-doctests-electra' of https://github.com/bhadreshpsa…
bhadreshpsavani Apr 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 13 additions & 2 deletions src/transformers/models/electra/modeling_electra.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1102,6 +1102,8 @@ def forward(
... 0
>>> ) # Batch size 1
>>> logits = model(input_ids).logits

@ydshieh ydshieh Apr 6, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be a good idea to use the example shown here

https://huggingface.co/google/electra-small-discriminator


```"""
return_dict = return_dict if return_dict is not None else self.config.use_return_dict

Expand Down Expand Up @@ -1175,6 +1177,9 @@ def set_output_embeddings(self, word_embeddings):
checkpoint=_CHECKPOINT_FOR_DOC,
output_type=MaskedLMOutput,
config_class=_CONFIG_FOR_DOC,
mask="[mask]",
expected_output="'togo'",
expected_loss=11.43,

@ydshieh ydshieh Apr 6, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To use Electra for mask LM task, we need checkpoints like

https://huggingface.co/google/electra-small-generator

You can find more with

https://huggingface.co/models?pipeline_tag=fill-mask&sort=downloads&search=electra

The

_CHECKPOINT_FOR_DOC = "google/electra-small-discriminator"

is not good for Mask LM, since it is

ELECTRA models are trained to distinguish "real" input tokens vs "fake" input tokens
generated by another neural network, similar to the discriminator of a [GAN]
(https://arxiv.org/pdf/1406.2661.pdf).

See https://huggingface.co/google/electra-small-discriminator

)
def forward(
self,
Expand Down Expand Up @@ -1256,9 +1261,11 @@ def __init__(self, config):
@add_start_docstrings_to_model_forward(ELECTRA_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
@add_code_sample_docstrings(
processor_class=_TOKENIZER_FOR_DOC,
checkpoint=_CHECKPOINT_FOR_DOC,
checkpoint="bhadresh-savani/electra-base-discriminator-finetuned-conll03-english ",

@ydshieh ydshieh Apr 6, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

output_type=TokenClassifierOutput,
config_class=_CONFIG_FOR_DOC,
expected_output="['B-LOC', 'B-ORG', 'O', 'O', 'O', 'O', 'O', 'B-LOC', 'O', 'B-LOC', 'I-LOC']",
expected_loss=0.11,
)
def forward(
self,
Expand Down Expand Up @@ -1336,9 +1343,11 @@ def __init__(self, config):
@add_start_docstrings_to_model_forward(ELECTRA_INPUTS_DOCSTRING.format("batch_size, sequence_length"))
@add_code_sample_docstrings(
processor_class=_TOKENIZER_FOR_DOC,
checkpoint=_CHECKPOINT_FOR_DOC,
checkpoint="deepset/electra-base-squad2",

@ydshieh ydshieh Apr 6, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

output_type=QuestionAnsweringModelOutput,
config_class=_CONFIG_FOR_DOC,
expected_output="'a nice puppet'",
expected_loss=0.0,
)
def forward(
self,
Expand Down Expand Up @@ -1592,6 +1601,8 @@ def forward(
>>> outputs = model(**inputs)

>>> prediction_logits = outputs.logits


@ydshieh ydshieh Apr 6, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can remove these extra new lines :-)

```"""
return_dict = return_dict if return_dict is not None else self.config.use_return_dict
if labels is not None:
Expand Down
1 change: 1 addition & 0 deletions utils/documentation_tests.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ src/transformers/models/blenderbot_small/modeling_blenderbot_small.py
src/transformers/models/convnext/modeling_convnext.py
src/transformers/models/data2vec/modeling_data2vec_audio.py
src/transformers/models/deit/modeling_deit.py
src/transformers/models/electra/modeling_electra.py
src/transformers/models/dpt/modeling_dpt.py
src/transformers/models/glpn/modeling_glpn.py
src/transformers/models/hubert/modeling_hubert.py
Expand Down