ElectraForMultipleChoice

[patil-suraj]

This PR add ElectraForMultipleChoice. One of the missing models in this project.

Since, for multiple choice pooled outputs are needed, added ElectraPooler class.

@sgugger , @LysandreJik

[codecov]

Codecov Report

Merging #4954 into master will increase coverage by 0.08%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #4954      +/-   ##
==========================================
+ Coverage   77.24%   77.33%   +0.08%     
==========================================
  Files         133      133              
  Lines       22134    22166      +32     
==========================================
+ Hits        17097    17141      +44     
+ Misses       5037     5025      -12     

Impacted Files	Coverage Δ
src/transformers/__init__.py	99.16% <ø> (ø)
src/transformers/modeling_auto.py	70.76% <ø> (ø)
src/transformers/configuration_electra.py	100.00% <100.00%> (ø)
src/transformers/modeling_electra.py	80.12% <100.00%> (+1.95%)	⬆️
src/transformers/modeling_utils.py	91.06% <0.00%> (+0.12%)	⬆️
src/transformers/trainer.py	39.62% <0.00%> (+0.23%)	⬆️
src/transformers/modeling_tf_utils.py	87.57% <0.00%> (+1.55%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update efeb75b...9cb71f9. Read the comment docs.

[sgugger]

Thanks for your PR! I think there is a way to make it more consistent with the other models for multiple choice and avoid introducing a new class, would you mind looking into it?

[sgugger]

Instead of introducing a new class, I'd rather use the existing SequenceSummary which can do the pool + dropout (see the XLNetForMultipleChoice for an example. It will require to add a few things to the config to make it work (see XLNetConfig for an example).

[patil-suraj]

I think there is a way to make it more consistent with the other models for multiple choice and avoid introducing a new class, would you mind looking into it?

Sure, I wasn't aware of SequenceSummary. I'll try to use it instead of Pooler

[stefan-it]

But couldn't you just use the ElectraClassificationHead for that pooling 🤔:

transformers/src/transformers/modeling_electra.py

Lines 346 to 362 in 86578bb

	class ElectraClassificationHead(nn.Module):
	"""Head for sentence-level classification tasks."""
	def __init__(self, config):
	super().__init__()
	self.dense = nn.Linear(config.hidden_size, config.hidden_size)
	self.dropout = nn.Dropout(config.hidden_dropout_prob)
	self.out_proj = nn.Linear(config.hidden_size, config.num_labels)
	def forward(self, features, **kwargs):
	x = features[:, 0, :] # take <s> token (equiv. to [CLS])
	x = self.dropout(x)
	x = self.dense(x)
	x = get_activation("gelu")(x) # although BERT uses tanh here, it seems Electra authors used gelu here
	x = self.dropout(x)
	x = self.out_proj(x)
	return x

[sgugger]

Sadly no, because it adds a linear layer to num_labels

[sgugger]

LGTM, @LysandreJik it's good for your review.

[LysandreJik]

LGTM, thanks @patil-suraj

[patil-suraj]

hi @LysandreJik could you tell me why these tests are failing now ? Thanks.

[sgugger]

Looks like @LysandreJik deleted a function by mistake during the merge (the create_and_check_electra_for_multiple_choice), could you add it back?

[LysandreJik]

My bad, sorry about that. Thanks for the fix!