Fix inverted conditional in TF common test!

[Rocketknight1]

Noticed a rather alarming conditional being backwards in the test_pt_tf_model_equivalence common test. This probably resulted in a lot of tests being skipped!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[Rocketknight1]

As expected this has raised a few bugs in the cross-test that were silent before - I'll see what I can do in this PR

[gante]

The change makes sense!

Re broken tests (which probably need to be fixed/skipped before merging) -- it means that the loss calculation has issues, correct?

[Rocketknight1]

Most likely - I'll investigate them all soon!

[Rocketknight1]

Quick summary of the fixes needed:

ESM: TFEsmForTokenClassification copied the computation from TFBertForTokenClassification, but this has some slightly odd BERT-specific behaviour and doesn't mask -100 in the same way as other models. Replaced it with the loss block from TFRobertaForTokenClassification and all tests pass.

GPT2: For model classes that take rank-3 inputs (e.g. MultipleChoice or DoubleHeads), when output_hidden_states=True , inputs have their second two dims flattened internally in the main model stem. This means that the output hidden_states are rank 3 (bsz, seq_len * num_choices, hidden_dim) and not rank 4 (bsz, num_choices, seq_len, hidden_dim). However, the PT model un-flattens the output for the final hidden_states, which means the last hidden state is rank-4, unlike the others which remain rank-3. In the old TF model, all hidden states are rank-3. I modified the TF code to un-flatten the last hidden state in the same way.

HUBERT: Loss computation especially for CTC overflows a lot with the default labels, which creates lots of inf values and makes it very hard to compare TF and PT losses. I skipped PT-TF equivalence testing for the losses, but keep it for all non-loss outputs.

Wav2Vec2: Same as HUBERT

XGLM: The PT XGLM model does a weird thing where it shifts labels by 1 and then adds pad_token_id as the final label to all samples. I'm not sure this is correct, but I modified the TF code to do the same. It's possible the TF code is the right one here though, in which case we should revert it and change the PT code instead.

[Rocketknight1]

@gante I fixed all the bugs that this surfaced, explained above ^

cc @sgugger for final review too

[sgugger]

Thanks a lot for fixing the condition in the base test and all the subsequent failures.

[ydshieh]

Thank you for the fix @Rocketknight1 ❤️ . And I apologize for the mistake I introduced ...