fix: suppress tokenizer parallelism warning in oneshot

[majiayu000]

SUMMARY:
Suppress the tokenizer parallelism warning that appears during oneshot calibration by setting TOKENIZERS_PARALLELISM=false in Oneshot.__init__.

The warning occurs when FastTokenizer's internal threading conflicts with dataset.map's multiprocessing (num_proc parameter). This fix sets the environment variable early in the oneshot lifecycle to prevent the conflict, while respecting any existing user-set value.

Closes #2007

TEST PLAN:

• Added unit tests in tests/llmcompressor/transformers/oneshot/test_tokenizer_parallelism.py
• Tests verify:
  1. TOKENIZERS_PARALLELISM is set to false when not already set
  2. Existing user-set TOKENIZERS_PARALLELISM values are respected
• All tests pass locally with pytest tests/llmcompressor/transformers/oneshot/test_tokenizer_parallelism.py -v

[gemini-code-assist]

Summary of Changes

Hello @majiayu000, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request aims to improve the user experience during oneshot calibration by eliminating a common tokenizer parallelism warning. It implements a targeted fix that automatically manages the TOKENIZERS_PARALLELISM environment variable to prevent conflicts arising from multiprocessing, while ensuring that any explicit user configurations are preserved. The change is supported by new unit tests to guarantee its intended functionality.

Highlights

• Suppress Tokenizer Parallelism Warning: The pull request addresses and suppresses a recurring tokenizer parallelism warning that appears during oneshot calibration. This is achieved by setting the TOKENIZERS_PARALLELISM environment variable to false within the Oneshot.__init__ method, specifically when it's not already set by the user. This prevents conflicts between FastTokenizer's internal threading and dataset.map's multiprocessing.
• Added Unit Tests: New unit tests have been introduced to verify the correct behavior of the TOKENIZERS_PARALLELISM environment variable handling. These tests confirm that the variable is set to false when not present and that any existing user-defined value for TOKENIZERS_PARALLELISM is respected and not overridden.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request effectively addresses the tokenizer parallelism warning by setting the TOKENIZERS_PARALLELISM environment variable. The implementation in Oneshot.__init__ is straightforward and correctly avoids overriding existing user settings. The new tests in test_tokenizer_parallelism.py are comprehensive, covering both scenarios where the environment variable is set and not set. I've provided a few suggestions to enhance the new test file's maintainability and robustness by introducing a constant for the environment variable name and strengthening the assertions.

[github-actions]

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

[kylesayrs]

Awesome investigation work! I think that now that we have a clear understanding of what's going on (which has now been documented), this change is justified. Please add a warning to make sure that users are aware of the change in environment.

[kylesayrs]

Thanks!

[mergify]

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

[brian-dellabetta]

Hi @majiayu000 , please run formatting as mergify explains and we can work to merge this in. Thanks for the contribution!

[majiayu000]

Done! I've run make style and all formatting checks have passed. The code is now ready for review. ✓

[mergify]

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

[HDCharles]

still failing, you need to do make style and make quality

[majiayu000]

@HDCharles All check passed. Thanks!

[mergify]

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

[HDCharles]

its still failing the quality checks

usually this happens when you have a different version of ruff

if you do pip install -e ./[dev]

or just make sure your ruff version is the same as in https://github.com/vllm-project/llm-compressor/blob/main/setup.py#L172 it should work.

[brian-dellabetta]

tests are green -- thanks for updating and for the contribution!