Reduce CPU-side per-token overhead in GenerateNextToken and SampleTopP

[hanbitmyths]

This PR is to reduce CPU-side per-token costs in two areas:

1. Cache per-token decisions at construction (generators.cpp, generators.h)

• Replace 3 per-token dynamic_cast<NemotronSpeechState*> with a one-time check at construction (is_nemotron_speech_model_)
• Pre-compute Phi3 ROPE rewind threshold and EP single-rope-factor flag, eliminating per-token string comparisons and device type checks
• Pre-determine sampling method (SamplingMethod enum + switch), replacing per-token conditional chain and moving parameter validation to construction

2. Eliminate O(V) exp() pass in SampleTopP (search.cpp)

• Use a tail-bound in FindNucleus: after partial-sorting K elements, bound the unsorted tail as (V-i-1) * exp(score_i) and terminate early when prefix_sum >= p * (prefix_sum + tail_bound)
• Start with K₀=16 (adopted from OpenVINO GenAI) with 4× geometric growth — minimizes wasted sort work when the top few tokens dominate, while avoiding O(V log V) full-sort fallback for flat distributions
• Exact-cutoff refinement for small vocabularies (V ≤ K₀ × growth_factor) where the tail bound is loose
• Saves ~V calls to std::exp() per token (128K-200K for production models)

Benchmarks (Batch=1, CPU, Release build)

SampleTopP (peaked distribution, p=0.95):

Vocab	Before (μs)	After (μs)	Speedup
32K	321	62	81%
128K	1119	272	76%
200K	1637	544	67%

Consistent 65-81% reduction across distribution shapes and top_p values (0.80-0.99).

Orchestration overhead (V=1000, greedy — isolates dispatch path):

	Before (μs)	After (μs)	Speedup
Greedy	0.41	0.30	27%

Adds two DISABLED_ benchmark tests for reproducibility.

[Copilot]

Pull request overview

This PR targets CPU-side per-token overhead reductions in the generation loop by (1) hoisting repeated per-token decisions (model/EP checks + sampling dispatch) into Generator construction, and (2) reducing Top-P sampling cost by avoiding an O(V) exp() pass in the nucleus-finding logic in search.cpp.

Changes:

• Cache per-token decisions in Generator (Nemotron speech state handling, Phi-3 ROPE rewind threshold, and sampling method selection/validation) to reduce per-token branching/dynamic checks.
• Update CPU Top-P sampling to use an adaptive partial-sort nucleus search with a log-space tail bound for early termination, eliminating the full-vocab softmax partition scan in common cases.
• Add disabled benchmarks to measure orchestration overhead and Top-P scaling across vocab sizes/distributions.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
src/generators.cpp	Adds per-generator initialization (InitPerTokenCache) and uses cached sampling dispatch + Phi-3 ROPE rewind threshold during GenerateNextToken.
src/generators.h	Introduces cached fields (SamplingMethod, ROPE threshold, EP/model flags) used to avoid per-token checks.
src/search.cpp	Reworks FindNucleus/SampleTopP to use a tail-bound early-termination strategy, removing the prior full-vocab exp-sum step in typical cases.
test/sampling_benchmark.cpp	Adds disabled benchmarks intended to isolate GenerateNextToken overhead and characterize Top-P scaling behavior.