fix: /pause_generation with --tokenizer-worker-num > 1

[lawrence-harmonic]

Motivation

Fix #21235

Modifications

We set the pause state in shared memory and poll it to update the local is_pause / is_pause_cond. Note that the consistency of is_pause will be reduced to eventual consistency in the multi-worker case. e.g. a pause immediately followed by a weight update may result in the weight update not seeing is_pause set immediately. Thus, we change the logic to continue checking for the pause flag while waiting for the writer lock.

The introduction of RWCondition is to achieve the following: we want to block others from changing is_pause during weight update, but not from polling is_pause.

NOTE: We do not fix the following existing issues:

• self.model_update_lock.writer_lock is process-local, thus does not actually ensure global exclusivity
• Similarly, /continue_generation can go to a different tokenizer worker and thus bypass the local condition variable lock.

In my opinion, the most correct way to do synchronization would be to have a system-wide model rwlock. Then, take reader locks in the schedulers instead of the tokenizers (maybe also take the writer locks in the schedulers instead of the tokenizers). The schedulers can release lock when paused. Then update weights could then just always take the model lock, simplifying the pause vs lock casework (skipping the lock, as we do now, technically works now but is a bit strange).

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[lawrence-harmonic]

/tag-and-rerun-ci

[lawrence-harmonic]

/tag-and-rerun-ci

[Qiaolin-Yu]

Thanks for the contribution! We will merge this in #24462 and have added you as a co-author.