Newcomer for help: If the same training corpus is used, is there a way to save the pre-tokenized data and load it directly next time?

[Wiselnn570]

Reminder

• I have read the README and searched the existing issues.

System Info

Currently, each time I fine-tune the program, it takes 6 hours to pre-tokenize, which is too time-consuming.
(p.s. I've tried launching in a streaming manner, but the GPU utilization is almost zero. The training time is even longer compared to offline preprocessing, which is not acceptable.)

Reproduction

[None]

Expected behavior

If the pre-tokenized data is saved locally, it can be loaded directly the next time for the same data, allowing to speed this process.

Others

No response

[hiyouga]

see https://github.com/hiyouga/LLaMA-Factory/tree/main/examples#preprocess-dataset