[Spec][Ngram] Support multiple SAMs with dynamic HTTP API#22203

Merged

hnyls2002 merged 29 commits intomainfrom

lsyin/multi-sam-http-api

Apr 7, 2026

Commits on Apr 6, 2026

move hash utils out of hicache_storage to break CUDA import chain
hnyls2002
committed
move hash functions to utils.py instead of separate hash_utils.py
hnyls2002
committed
support multiple SAMs with dynamic HTTP API
hnyls2002
committed
auto-generate corpus_id when not provided
hnyls2002
committed
add multi-SAM and mock scheduler tests
hnyls2002
committed
replace mock scheduler tests with HTTP endpoint tests
hnyls2002
committed
use unique_ptr for SAM ownership, restore mutex comments
hnyls2002
committed
reject empty corpus, remove active_sams indirection
hnyls2002
committed
remove startup corpus loading, all SAMs registered via HTTP API
hnyls2002
committed
restore startup corpus preloading via --speculative-ngram-external-corpus-path, reusing add_external_corpus
hnyls2002
committed
async corpus loading: background thread build, non-blocking scheduler
hnyls2002
committed
remove staging_corpus_id_ member, pass corpus_id to finish instead
hnyls2002
committed
add FIXME for single staging slot limitation
hnyls2002
committed
guard against concurrent corpus loads in startExternalCorpusLoad
hnyls2002
committed
extract ExternalCorpusManager from scheduler
hnyls2002
committed
restore startup corpus path validation
hnyls2002
committed
restore validation tests for startup corpus args
hnyls2002
committed
use newline delimiter for corpus ID list to avoid comma-in-id bug
hnyls2002
committed
remove no-op staging_sam_.reset() and redundant lock.unlock()
hnyls2002
committed
truncate documents exceeding max_tokens instead of loading unbounded
hnyls2002
committed
add thread.join() for memory visibility guarantee in check_pending_load
hnyls2002
committed
validate request body in add_external_corpus endpoint, remove redundant init
hnyls2002
committed
hoist loop-invariant budget computation out of batchMatch loop
hnyls2002
committed
initialize external_corpus_manager to None when spec decoding is disabled
hnyls2002
committed
skip TestMultiSamHttpMock on CPU-only runners lacking CUDA libraries
hnyls2002
committed
fix tests calling removed load_external_corpus method
hnyls2002
committed
use file path as corpus_id in file-based tests
hnyls2002
committed

Commits on Apr 7, 2026

remove __default__ corpus_id convention from tests
hnyls2002
committed
Merge branch 'main' into lsyin/multi-sam-http-api
hnyls2002
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Spec][Ngram] Support multiple SAMs with dynamic HTTP API#22203

[Spec][Ngram] Support multiple SAMs with dynamic HTTP API#22203
hnyls2002 merged 29 commits intomainfrom
lsyin/multi-sam-http-api

Commits on Apr 6, 2026

move hash utils out of hicache_storage to break CUDA import chain

move hash functions to utils.py instead of separate hash_utils.py

support multiple SAMs with dynamic HTTP API

auto-generate corpus_id when not provided

add multi-SAM and mock scheduler tests

replace mock scheduler tests with HTTP endpoint tests

use unique_ptr for SAM ownership, restore mutex comments

reject empty corpus, remove active_sams indirection

remove startup corpus loading, all SAMs registered via HTTP API

restore startup corpus preloading via --speculative-ngram-external-corpus-path, reusing add_external_corpus

async corpus loading: background thread build, non-blocking scheduler

remove staging_corpus_id_ member, pass corpus_id to finish instead

add FIXME for single staging slot limitation

guard against concurrent corpus loads in startExternalCorpusLoad

extract ExternalCorpusManager from scheduler

restore startup corpus path validation

restore validation tests for startup corpus args

use newline delimiter for corpus ID list to avoid comma-in-id bug

remove no-op staging_sam_.reset() and redundant lock.unlock()

truncate documents exceeding max_tokens instead of loading unbounded

add thread.join() for memory visibility guarantee in check_pending_load

validate request body in add_external_corpus endpoint, remove redundant init

hoist loop-invariant budget computation out of batchMatch loop

initialize external_corpus_manager to None when spec decoding is disabled

skip TestMultiSamHttpMock on CPU-only runners lacking CUDA libraries

fix tests calling removed load_external_corpus method

use file path as corpus_id in file-based tests

Commits on Apr 7, 2026

remove default corpus_id convention from tests

Merge branch 'main' into lsyin/multi-sam-http-api