ERROR: Failed building wheel for tokenizers

[RyanFDing]

Trying to follow the instructions given in section 0 (prerequisites) for running the fine-tuned models in x-fact:
everything works except this line (pip install --editable ./)

I've tried many things, such as installing the rust language and adding vs build tools, but I keep getting this error:

  error[E0557]: feature has been removed
        --> C:\Users\ding\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lock_api-0.3.4\src\lib.rs:91:42
         |
      91 | #![cfg_attr(feature = "nightly", feature(const_fn))]
         |                                          ^^^^^^^^ feature has been removed
         |
         = note: split into finer-grained feature gates

      For more information about this error, try `rustc --explain E0557`.
      error: could not compile `lock_api` (lib) due to 1 previous error

      Caused by:
        process didn't exit successfully: `rustc --crate-name lock_api --edition=2018 C:\Users\ding\.cargo\registry\src\index.crates.io-6f17d22bba15001f\lock_api-0.3.4\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata,link -C opt-level=3 -C embed-bitcode=no --cfg "feature=\"nightly\"" --check-cfg cfg(docsrs) --check-cfg "cfg(feature, values(\"nightly\", \"owning_ref\", \"serde\"))" -C metadata=e3b4e41dabc89c13 -C extra-filename=-e3b4e41dabc89c13 --out-dir C:\Users\ding\AppData\Local\Temp\pip-install-d5vow5o_\tokenizers_3feb35558e4a4c8e8d80cedaebf0e349\target\release\deps -L dependency=C:\Users\ding\AppData\Local\Temp\pip-install-d5vow5o_\tokenizers_3feb35558e4a4c8e8d80cedaebf0e349\target\release\deps --extern scopeguard=C:\Users\ding\AppData\Local\Temp\pip-install-d5vow5o_\tokenizers_3feb35558e4a4c8e8d80cedaebf0e349\target\release\deps\libscopeguard-a6a56faef7277818.rmeta --cap-lints allow` (exit code: 1)
      warning: build failed, waiting for other jobs to finish...
      error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib --` failed with code 101
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

I've tried many fixes and it always gets a little further but still gets stuck on the wheel.

[ashim95]

Hi @RyanFDing
Sorry for the late reply. And thanks for pointing this issue.

Just now, I have tested the code on a linux machine with python 3.6.8 and it works fine. Following are the requirements:

certifi==2024.8.30
charset-normalizer==2.0.12
click==8.0.4
dataclasses==0.8
filelock==3.4.1
idna==3.10
importlib-metadata==4.8.3
importlib-resources==5.4.0
joblib==1.1.1
numpy==1.19.5
packaging==21.3
protobuf==4.21.0
pyparsing==3.1.4
regex==2023.8.8
requests==2.27.1
sacremoses==0.0.53
scikit-learn==0.24.2
scipy==1.5.4
sentencepiece==0.2.0
six==1.16.0
tensorboardX==2.6.2.2
threadpoolctl==3.1.0
tokenizers==0.8.1rc1
torch==1.7.1
tqdm==4.64.1
-e git+https://github.com/utahnlp/x-fact@01bb640b5c352e3e3154ccc9e961df76f33dd397#egg=transformers&subdirectory=transformers
typing_extensions==4.1.1
urllib3==1.26.20
zipp==3.6.0

If you are still facing this issue, can you try following the above instructions and version specifications. Please let me know if this doesn't work for you.

-- ashim