Improve Dataset Processing Multiprocessing, Sharding, and Qwen Tokenizer Bug Fix.#2918

Merged

winglian merged 10 commits into

axolotl-ai-cloud:mainfrom

VarunGumma:main

Jul 17, 2025

Commits on Jul 14, 2025

Added a feature to save prepared dataset in specified shards, removed limiter on multiprocessing during tokenization, and a bug fix of qwen tokenizer
VarunGumma
committed
Merge branch 'axolotl-ai-cloud:main' into main
VarunGumma
authored
Merge branch 'axolotl-ai-cloud:main' into main
VarunGumma
authored

Commits on Jul 15, 2025

Merge branch 'axolotl-ai-cloud:main' into main
VarunGumma
authored
removed limiters and fixed config variable name
VarunGumma
committed
black lint
VarunGumma
committed

Commits on Jul 16, 2025

fixed process_count
VarunGumma
committed

Commits on Jul 17, 2025