Incorrect assertion in pipeline test test_dbmdz_english()

[davidbenton]

System Info

• transformers version: 4.22.0.dev0
• Platform: macOS-12.4-x86_64-i386-64bit
• Python version: 3.9.13
• Huggingface_hub version: 0.8.1
• PyTorch version (GPU?): 1.11.0 (False)
• Tensorflow version (GPU?): 2.9.1 (False)
• Flax version (CPU?/GPU?/TPU?): 0.5.2 (cpu)
• Jax version: 0.3.6
• JaxLib version: 0.3.5
• Using GPU in script?: n
• Using distributed or parallel set-up in script?: n

Who can help?

@Narsil

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

RUN_SLOW=1 RUN_PIPELINE_TESTS=yes pytest tests/pipelines/test_pipelines_token_classification.py::TokenClassificationPipelineTests::test_dbmdz_english

Fails with two notable diffs: the "UN" entity offsets in the assertion don't match the offsets in the input string itself (off by two characters), and the index doesn't match. Output:

======================================= FAILURES ========================================
__________________ TokenClassificationPipelineTests.test_dbmdz_english __________________

self = <tests.pipelines.test_pipelines_token_classification.TokenClassificationPipelineTests testMethod=test_dbmdz_english>

    @require_torch
    @slow
    def test_dbmdz_english(self):
        # Other sentence
        NER_MODEL = "dbmdz/bert-large-cased-finetuned-conll03-english"
        model = AutoModelForTokenClassification.from_pretrained(NER_MODEL)
        tokenizer = AutoTokenizer.from_pretrained(NER_MODEL, use_fast=True)
        sentence = """Enzo works at the UN"""
        token_classifier = pipeline("ner", model=model, tokenizer=tokenizer)
        output = token_classifier(sentence)
>       self.assertEqual(
            nested_simplify(output),
            [
                {"entity": "I-PER", "score": 0.997, "word": "En", "start": 0, "end": 2, "index": 1},
                {"entity": "I-PER", "score": 0.996, "word": "##zo", "start": 2, "end": 4, "index": 2},
                {"entity": "I-ORG", "score": 0.999, "word": "UN", "start": 22, "end": 24, "index": 7},
            ],
        )
E       AssertionError: Lists differ: [{'en[24 chars] 0.998, 'index': 1, 'word': 'En', 'start': 0, [179 chars] 20}] != [{'en[24 chars] 0.997, 'word': 'En', 'start': 0, 'end': 2, 'i[179 chars]: 7}]
E       
E       First differing element 0:
E       {'ent[15 chars]'score': 0.998, 'index': 1, 'word': 'En', 'start': 0, 'end': 2}
E       {'ent[15 chars]'score': 0.997, 'word': 'En', 'start': 0, 'end': 2, 'index': 1}
E       
E         [{'end': 2,
E           'entity': 'I-PER',
E           'index': 1,
E       -   'score': 0.998,
E       ?                ^
E       
E       +   'score': 0.997,
E       ?                ^
E       
E           'start': 0,
E           'word': 'En'},
E          {'end': 4,
E           'entity': 'I-PER',
E           'index': 2,
E       -   'score': 0.997,
E       ?                ^
E       
E       +   'score': 0.996,
E       ?                ^
E       
E           'start': 2,
E           'word': '##zo'},
E       -  {'end': 20,
E       ?           ^
E       
E       +  {'end': 24,
E       ?           ^
E       
E           'entity': 'I-ORG',
E       -   'index': 6,
E       ?            ^
E       
E       +   'index': 7,
E       ?            ^
E       
E           'score': 0.999,
E       -   'start': 18,
E       ?            ^^
E       
E       +   'start': 22,
E       ?            ^^
E       
E           'word': 'UN'}]

tests/pipelines/test_pipelines_token_classification.py:284: AssertionError

Expected behavior

[a green dot]

[Narsil]

@davidbenton
Thanks for the info, I was able to reproduce on torch 1.11.0 but this is fixed on 1.12.0.

We're aware of some modifications within the nn.Linear between those two versions. The errors are relatively minor actually, but since this is a random model (meaning not trained on real data) it's much more sensitive to those tiny fluctuations.
That's why the test fails on 1.11.0 while it works on 1.12.0.

EDIT:
I though I had tested the test and it worked on 1.12.0 but if my explanation was correct, then a diff should have had occurred on the test itself.
I found out that this commit touch the file: 95113d1

This commit actually changed the string, and makes this slow test fail.
I am guessing this is an automated change. Looked at the diff, it seems only this slow test was affected.
We can either rollback the string, or update the values.

So the 1.11.0 vs 1.12.0 seems like it doesn't explain the difference here.

@ydshieh Maybe can you confirm ?

[davidbenton]

Hmm, it fails for me on 1.12.0 also (CPU, env otherwise same as above). How could end: 24 be correct for a string with length 20? Those offsets should be in sentence indices, right?

I see you're on the track now.

[Narsil]

Pinging @sgugger too here.

Thanks for reporting @davidbenton !

[ydshieh]

Since that commit, we do have this failure in Slack CI report. From the changes, I think it try to fix all the the. So reverting the change on the expected value is good to me.

Thank you, @davidbenton

[sgugger]

Yes, it slipped through the crack as the contributor was trying to fix typos and I didn't pay attention this one was intentional.