Skip to content
This repository has been archived by the owner on Oct 9, 2023. It is now read-only.

Loading pretrained translation model fails due to Not found: "/Users/ethan/..." #334

Closed
akihironitta opened this issue May 23, 2021 · 1 comment · Fixed by #332
Closed
Assignees
Labels
bug / fix Something isn't working help wanted Extra attention is needed

Comments

@akihironitta
Copy link
Contributor

🐛 Bug

The current pretrained model (https://flash-weights.s3.amazonaws.com/translation_model_en_ro.pt) somehow includes a property (?) that only works in a certain environment, i.e. "/Users/ethan/...".

$ strings ~/.cache/torch/hub/checkpoints/translation_model_en_ro.pt|grep "/Users/ethan"
/Users/ethan/Documents/grid/lightning-flash/flash_examples/finetuning/lightning_logs/version_169/checkpoints/epoch=2-step=59.ckptr
/Users/ethan/Documents/grid/lightning-flash/flash_examples/finetuning/lightning_logs/version_169/checkpointsr
/Users/ethan/.cache/huggingface/transformers/1954839338efdfd7d7cd66bb676626f3fa5404a96e4c3fd9e0d777f08fb2f7fb.2a52a89183278824d0afa78f92263dc46ca0c520a60df7071b6ae4e00fc5bef7r
/Users/ethan/.cache/huggingface/transformers/1e4b193c82cb1b7fe168ad6685f3a053a2b1ab3482d2d12c306282e03ff379f0.afd22f83f2170348e28be1f817974819f568155c58acd1bb3800cd7937db36a1r
/Users/ethan/.cache/huggingface/transformers/1954839338efdfd7d7cd66bb676626f3fa5404a96e4c3fd9e0d777f08fb2f7fb.2a52a89183278824d0afa78f92263dc46ca0c520a60df7071b6ae4e00fc5bef7r
/Users/ethan/.cache/huggingface/transformers/1e4b193c82cb1b7fe168ad6685f3a053a2b1ab3482d2d12c306282e03ff379f0.afd22f83f2170348e28be1f817974819f568155c58acd1bb3800cd7937db36a1r
/Users/ethan/.cache/huggingface/transformers/1954839338efdfd7d7cd66bb676626f3fa5404a96e4c3fd9e0d777f08fb2f7fb.2a52a89183278824d0afa78f92263dc46ca0c520a60df7071b6ae4e00fc5bef7r
/Users/ethan/.cache/huggingface/transformers/1e4b193c82cb1b7fe168ad6685f3a053a2b1ab3482d2d12c306282e03ff379f0.afd22f83f2170348e28be1f817974819f568155c58acd1bb3800cd7937db36a1r
/Users/ethan/.cache/huggingface/transformers/1954839338efdfd7d7cd66bb676626f3fa5404a96e4c3fd9e0d777f08fb2f7fb.2a52a89183278824d0afa78f92263dc46ca0c520a60df7071b6ae4e00fc5bef7r
/Users/ethan/.cache/huggingface/transformers/1e4b193c82cb1b7fe168ad6685f3a053a2b1ab3482d2d12c306282e03ff379f0.afd22f83f2170348e28be1f817974819f568155c58acd1bb3800cd7937db36a1r

To Reproduce

from flash.text import TranslationTask
model = TranslationTask.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/translation_model_en_ro.pt")

and here's the output:

Traceback (most recent call last):
  File "/__w/1/s/flash_examples/predict/translation.py", line 23, in <module>
    model = TranslationTask.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/translation_model_en_ro.pt")
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/pytorch_lightning/core/saving.py", line 135, in load_from_checkpoint
    checkpoint = pl_load(checkpoint_path, map_location=lambda storage, loc: storage)
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/cloud_io.py", line 30, in load
    return torch.hub.load_state_dict_from_url(str(path_or_url), map_location=map_location)
  File "/usr/local/lib/python3.8/dist-packages/torch/hub.py", line 559, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 594, in load
    return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.8/dist-packages/torch/serialization.py", line 853, in _load
    result = unpickler.load()
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/transformers/models/marian/tokenization_marian.py", line 286, in __setstate__
    self.spm_source, self.spm_target = (load_spm(f) for f in self.spm_files)
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/transformers/models/marian/tokenization_marian.py", line 286, in <genexpr>
    self.spm_source, self.spm_target = (load_spm(f) for f in self.spm_files)
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/transformers/models/marian/tokenization_marian.py", line 313, in load_spm
    spm.Load(path)
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/sentencepiece/__init__.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/home/AzDevOps_azpcontainer/.local/lib/python3.8/site-packages/sentencepiece/__init__.py", line 171, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
OSError: Not found: "/Users/ethan/.cache/huggingface/transformers/1954839338efdfd7d7cd66bb676626f3fa5404a96e4c3fd9e0d777f08fb2f7fb.2a52a89183278824d0afa78f92263dc46ca0c520a60df7071b6ae4e00fc5bef7": No such file or directory Error #2

https://dev.azure.com/PytorchLightning/lightning%20Flash/_build/results?buildId=9579&view=logs&j=cb44143e-1b72-550b-97cf-26ffc4f8d032&t=99903360-f385-53aa-153f-fd44167f075f&l=243

Expected behavior

Successfully running the following lines.

from flash.text import TranslationTask
model = TranslationTask.load_from_checkpoint("https://flash-weights.s3.amazonaws.com/translation_model_en_ro.pt")

Environment

  • PyTorch Version (e.g., 1.0):
  • OS (e.g., Linux):
  • How you installed PyTorch (conda, pip, source):
  • Build command you used (if compiling from source):
  • Python version:
  • CUDA/cuDNN version:
  • GPU models and configuration:
  • Any other relevant information:
@akihironitta akihironitta added bug / fix Something isn't working help wanted Extra attention is needed and removed help wanted Extra attention is needed labels May 23, 2021
@ethanwharris
Copy link
Collaborator

Yeah, this is an issue with pickle, somehow our checkpoints are pickled by the tokenizer. We need to figure out a way to stop it as anything using SentencePiece will have this problem

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug / fix Something isn't working help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants