Error when initializing from the transformers pipeline

[EtienneGagnon1]

Hello,

I get an error when trying to initialize models that rely on your tokenizer from the transformers package's pipeline. Here is code that yields the error as well as the traceback.

from transformers import pipeline 

sentiment_analyzer = pipeline(
    "sentiment-analysis", model="cl-tohoku/bert-base-japanese", tokenizer="cl-tohoku/bert-base-japanese")

Traceback (most recent call last):
  File "<input>", line 3, in <module>
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\pipelines\__init__.py", line 377, in pipeline
    tokenizer = AutoTokenizer.from_pretrained(tokenizer, revision=revision, use_fast=use_fast)
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 391, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 294, in tokenizer_class_from_name
    if c.__name__ == class_name:
AttributeError: 'NoneType' object has no attribute '__name__'

[prtsk]

I have the same issue when calling the tokenizer. It seems like the AutoTokenizer is reading the path to the tokenizer as empty. THis problem persists even when I download the model.

[singletongue]

Thank you for raising the issue. Unfortunately, I was not able to reproduce the error with transformers==4.3.3.

>>> from transformers import pipeline
>>> sentiment_analyzer = pipeline(
...     "sentiment-analysis", model="cl-tohoku/bert-base-japanese", tokenizer="cl-tohoku/bert-base-japanese")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 445M/445M [00:40<00:00, 11.1MB/s]
Some weights of the model checkpoint at cl-tohoku/bert-base-japanese were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at cl-tohoku/bert-base-japanese and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
>>>

Is your installed transformers library up-to-date?
If so, would you please delete the cache of the downloaded models and tokenizers and try again?

$ rm -r ~/.cache/huggingface/transformers

Thank you.

[EtienneGagnon1]

Hello thank you for your reply.

I did update my package to version 4.3.3 and cleared the cache but I am still getting the same error:
my python version is 3.8.8 I also tested with python 3.7.

import transformers
from transformers import pipeline

print(transformers.__version__)

sentiment_analyzer = pipeline('sentiment-analysis',
                              model='cl-tohoku/bert-base-japanese',
                              tokenizer='cl-tohoku/bert-base-japanese')

python analyze_sentiment.py
4.3.3
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 479/479 [00:00<00:00, 481kB/s]
Downloading: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 445M/445M [00:38<00:00, 11.7MB/s]
Traceback (most recent call last):
  File "analyze_sentiment.py", line 8, in <module>
    tokenizer='cl-tohoku/bert-base-japanese')
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\pipelines\__init__.py", line 373, in pipeline
    tokenizer = AutoTokenizer.from_pretrained(tokenizer, revision=revision, use_fast=use_fast)
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 370, in from_pretrained
    tokenizer_class = tokenizer_class_from_name(tokenizer_class_candidate)
  File "C:\Users\gagno\Anaconda3\envs\japanese_admin_scrape\lib\site-packages\transformers\models\auto\tokenization_auto.py", line 273, in tokenizer_class_from_name
    if c.__name__ == class_name:
AttributeError: 'NoneType' object has no attribute '__name__'

[prtsk]

Yes! I followed your suggestion, but I still get the same error.

[singletongue]

Is your installed tokenizers library up-to-date? (it may not necessarily be updated along with the transformers library.)
If there's still an error, would you please try specifying use_fast=False when initializing a pipeline?
Thank you.

[EtienneGagnon1]

Hello, thank you for your help! The tokenizer package was up to date (0.10.1). the the use_fast=False argument solved the issue.

[singletongue]

Now I'm able to reproduce the error. It seems that the error is raised if sentencepiece is not installed.
Although sentencepiece is not required for using BertJapaneseTokenizer, the absence of this package is causing an exception in the process of initializing a AutoTokenizer.

I will make a PR to the transformers repository, but for now please install sentencepiece to avoid this issue.

$ pip install sentencepiece

Thanks!