Support fine-grained control of requests that are run together

[fzyzcjy]

Motivation

Currently, when submitting requests (e.g. engine.generate or HTTP call), we have no control which requests will be run together in a single batch and which will not, partially because the intrinsic undeterminism of IPC. However, in some scenarios, it would be great if we have more control. For example:

• Benchmarking and profiling (e.g. we want to know the behavior when exactly having "1024 token x 8 req per GPU"; this is the primary reason why I made this PR)
• Testing (e.g. in two-batch-overlap, we may want to test when "one card has 2 req while another card has 1 req, it should be disabled")

Thus this PR adds this feature. Since it is only used for benchmarking or testing, the code is not efficient (e.g. it calls torch.distributed that may be reducible to some extent), and may have rough edges.

Modifications

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[fzyzcjy]

Ping me when this PR is to be merged - currently I only resolve conflicts in #4068, and will port the resolve code back here when pinged.