[Spec][Ngram] Add output-as-corpus accept length benchmark for external SAM

[hnyls2002]

Motivation

Part of Ngram refactoring series #21052
Following #22203

No end-to-end test proving that external SAM actually improves speculative decoding accept length.

Modifications

Add test_output_as_corpus_boosts_accept_length to TestNgramSpeculativeDecodingFlashinfer:

1. Generate outputs with temperature=0 (baseline, trie only), record avg_spec_accept_length
2. POST /add_external_corpus with generated outputs as corpus
3. Regenerate same prompts, assert SAM accept length >= 2x baseline

Single server launch — reuses existing flashinfer test class with --speculative-ngram-external-sam-budget 8. Also removes redundant TestNgramExternalSamSmoke (SAM functionality now covered by this benchmark + unit tests).

[gemini-code-assist]

Code Review

This pull request adds the TestNgramExternalSamAcceptLength test class to verify that N-gram speculative decoding with an external corpus significantly increases acceptance length. Feedback includes improving the robustness of server statistics retrieval for multi-GPU environments and a minor refactor of the baseline output generation loop.

[gemini-code-assist]

The _get_accept_length method assumes that internal_states is non-empty and that the first entry contains the avg_spec_accept_length key. While this is likely true for a single-GPU setup, it could fail with a KeyError or IndexError in multi-GPU (DP/PP) environments where stats might be distributed or not yet reported. Consider making this more robust by checking for the key's existence or averaging across all ranks.

Suggested change

	def _get_accept_length(self, base_url):
	server_info = requests.get(base_url + "/get_server_info").json()
	return server_info["internal_states"][0]["avg_spec_accept_length"]
	def _get_accept_length(self, base_url):
	server_info = requests.get(base_url + "/get_server_info").json()
	states = server_info.get("internal_states", [])
	accept_lengths = [s.get("avg_spec_accept_length", 0.0) for s in states if "avg_spec_accept_length" in s]
	return sum(accept_lengths) / len(accept_lengths) if accept_lengths else 0.0

[gemini-code-assist]

In the baseline generation loop, generated_outputs is overwritten in each round. While temperature=0 should produce deterministic results, it is generally safer to either accumulate all outputs or explicitly use the last round's results to ensure the corpus is fully populated as intended.

Suggested change

	generated_outputs = []
	for _ in range(self.num_rounds):
	outputs = self._generate_batch(
	self.base_url, self.prompts, self.max_new_tokens
	)
	generated_outputs = outputs
	generated_outputs = []
	for _ in range(self.num_rounds):
	generated_outputs = self._generate_batch(
	self.base_url, self.prompts, self.max_new_tokens
	)

[hnyls2002]

/rerun-test registered/spec/test_ngram_speculative_decoding.py

[github-actions]

✅ 1-gpu-h100 (1 test): View workflow run

cd test/ && python3 registered/spec/test_ngram_speculative_decoding.py