Skip to content

Conversation

@ydshieh
Copy link
Collaborator

@ydshieh ydshieh commented Oct 5, 2023

What does this PR do?

Doc builder docker image build starts to fail with

The package you are trying to install is only a placeholder project on PyPI.org repository.
This package is hosted on NVIDIA Python Package Index.
  
This package can be installed as:

$ pip install --no-cache-dir --extra-index-url https://pypi.nvidia.com/ pytorch-quantization

I tried to install it with the suggested command, but the doc build step will fail with some cuda libary issue.
(when building transformers/docs/source/en/model_doc/qdqbert.md)

I removed the line that installs pytorch-quantization and doc build can pass (and docker image built).

@ydshieh ydshieh changed the title Fix docker build 003 Don't install pytorch-quantization in Doc Builder docker file Oct 5, 2023
@ydshieh ydshieh force-pushed the fix_docker_build_003 branch from 316e7ab to a13a6aa Compare October 5, 2023 14:12
RUN python3 -m pip install --no-cache-dir ./transformers[deepspeed]

RUN python3 -m pip install --no-cache-dir torchvision git+https://github.com/facebookresearch/detectron2.git pytesseract
RUN python3 -m pip install --no-cache-dir pytorch-quantization --extra-index-url https://pypi.ngc.nvidia.com
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know if this will affect the doc build for the model QDQBert. I tested it locally, the doc build completed without failure, including transformers/docs/source/en/model_doc/qdqbert.md.

@ydshieh ydshieh requested a review from LysandreJik October 5, 2023 14:24
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Oct 5, 2023

The documentation is not available anymore as the PR was closed or merged.

@ydshieh
Copy link
Collaborator Author

ydshieh commented Oct 5, 2023

For the record, here is the issue when running the suggested installation command: at the doc build time

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1282, in _get_module
    return importlib.import_module("." + module_name, self.__name__)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/transformers/models/qdqbert/modeling_qdqbert.py", line 60, in <module>
    from pytorch_quantization import nn as quant_nn
  File "/usr/local/lib/python3.8/site-packages/pytorch_quantization/__init__.py", line 20, in <module>
    from .quant_modules import *
  File "/usr/local/lib/python3.8/site-packages/pytorch_quantization/quant_modules.py", line 23, in <module>
    from pytorch_quantization import nn as quant_nn
  File "/usr/local/lib/python3.8/site-packages/pytorch_quantization/nn/__init__.py", line 19, in <module>
    from pytorch_quantization.nn.modules.tensor_quantizer import *
  File "/usr/local/lib/python3.8/site-packages/pytorch_quantization/nn/modules/tensor_quantizer.py", line 24, in <module>
    from pytorch_quantization.tensor_quant import QuantDescriptor, tensor_quant, fake_tensor_quant, scaled_e4m3
  File "/usr/local/lib/python3.8/site-packages/pytorch_quantization/tensor_quant.py", line 28, in <module>
    from pytorch_quantization import cuda_ext
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 197, in build_mdx_files
    content, new_anchors, source_files, errors = resolve_autodoc(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 123, in resolve_autodoc
    doc = autodoc(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/autodoc.py", line 474, in autodoc
    obj = find_object_in_package(object_name=object_name, package=package)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/autodoc.py", line 39, in find_object_in_package
    submodule = getattr(module, split, None)
  File "/usr/local/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1273, in __getattr__
    value = getattr(module, name)
  File "/usr/local/lib/python3.8/site-packages/transformers/utils/import_utils.py", line 1272, in __getattr__
    module = self._get_module(self._class_to_module[name])
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/doc-builder", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/doc_builder_cli.py", line 47, in main
    args.func(args)
  File "/usr/local/lib/python3.8/site-packages/doc_builder/commands/build.py", line 102, in build_command
    build_doc(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 365, in build_doc
    anchors_mapping, source_files_mapping = build_mdx_files(
  File "/usr/local/lib/python3.8/site-packages/doc_builder/build_doc.py", line 230, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
RuntimeError: There was an error when converting transformers/docs/source/en/model_doc/qdqbert.md to the MDX format.
Failed to import transformers.models.qdqbert.modeling_qdqbert because of the following error (look up to see its traceback):
libcudart.so.11.0: cannot open shared object file: No such file or directory

Copy link
Member

@LysandreJik LysandreJik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LysandreJik
Copy link
Member

Thanks Yih-Dar!

@ydshieh ydshieh merged commit 75a33d6 into main Oct 5, 2023
@ydshieh ydshieh deleted the fix_docker_build_003 branch October 5, 2023 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants