tests: add model resolution test on synthetic repo listings by ServeurpersoCom · Pull Request #26172 · ggml-org/llama.cpp

ServeurpersoCom · 2026-07-27T11:28:51Z

Overview

Adding the synthetic repo tests we discussed on #26165, download.cpp and arg.cpp are included inside a namespace with hf_cache monkey patched to serve hardcoded listings mimicking real vendor conventions, so the resolution and the model handler assembly run end-to-end through the real CLI parsing on permuted listings, no code modified and no network needed, dspark will just add its keyword, a fixture line and a table line

The test harness is table-driven so new layouts or resolution rules can be added with a fixture line and a table line

Additional information

Follow-up of #26165

Requirements

I have read and agree with the contributing guidelines
AI usage disclosure: YES

ServeurpersoCom · 2026-07-28T15:35:26Z

Ouch, I'm fixing CI

ggerganov

Seems quite a bit hacky and difficult for human to follow. But it's probably better than nothing. Let's have @ngxson take a look too.

ServeurpersoCom · 2026-07-28T18:01:47Z

Yes, it's just an idea that's easy to refactor using AI, but we could also create a real repo with dummy mini-files, though that would involve a small internet request.

I actually started with the other approach, exposing the resolution and the task assembly as functions taking the listing as input, which gives a trivial readable test at the cost of a small API addition. The real solution would be to refactor the file resolution into a clean reusable unit, but I am not sure it is worth it.

Current approach is that it runs the real production code path end-to-end, real CLI parsing and real handler init, so it will warn if we break actual model downloads, while exposed functions would only test what we choose to expose.

ggerganov · 2026-07-28T18:08:53Z

Yes, let's see if this approach would work well enough. We can reconsider later if we still encounter problems.

ServeurpersoCom · 2026-07-28T18:12:25Z

Yes, it acts as a canary that will break on the first refactor, which means the refactor will have to take testability into account, and this hack can then be removed.

ServeurpersoCom · 2026-07-28T21:06:52Z

Refactor way look like :

Testable heuristic 33bf0c6
Test unit c63f955

ngxson · 2026-07-28T21:20:32Z

+std::pair<long, std::vector<char>> common_remote_get_content(const std::string & url, const common_remote_params & params);
+std::pair<std::string, std::string> common_download_split_repo_tag(const std::string & hf_repo_with_tag);
+void common_download_run_tasks(const std::vector<common_download_task> & tasks);
+std::vector<std::string> common_download_get_all_parts(const std::string & url);
+std::vector<common_cached_model_info> common_list_cached_models();
+int common_download_file_single(const std::string & url, const std::string & path, const common_download_opts & opts = {}, bool skip_etag = false);
+std::string common_docker_resolve_model(const std::string & docker);
+bool common_download_remove(const std::string & hf_repo_with_tag);
+common_download_hf_plan common_download_get_hf_plan(const common_params_model & model, const common_download_opts & opts);
+


I think this test case is useful, but we should find a way to avoid all of the boilerplate on the top of this file

I assume this boilerplate is to avoid making network calls. wondering if we can just stub the httplib:Client by making a derived class from it. will quickly test if that's even possible or not.

btw, can you confirm if there are any other reasons why we need this long boilerplate?

I just made the cleaner alternative PTAL #26247

About the boilerplate: the network avoidance is the small part (the 20 line fake hf_cache, the natural seam, httplib::Client would not intercept the cached listings). The rest is the price of compiling namespaced copies of the sources: ADL ambiguity with the global declarations forces the rename macros and the verbatim redeclarations for the fatal warnings CI, plus the unused pragmas, the winsock ordering and the explicit cpp-httplib link under lld-link.

But given that auto downloading is an essential component of the user experience, and that the other modalities (vocoder etc...) will bring additional cases, I think the refactor is worth it.

ngxson · 2026-07-28T21:53:02Z

Ok so let's wrap httplib::Client around a new struct common_http_client, and provide a derived stub class at test time. I partially done the work here, feel free to pick it up: master...xsn/demo_common_http_client

ngxson

much cleaner now

ngxson · 2026-07-29T10:31:45Z

+#define REQUIRE(x) do {                                                         \
+    if (!(x)) {                                                                 \
+        fprintf(stderr, "%s:%d: [%s] REQUIRE(%s) failed\n",                     \
+                __FILE__, __LINE__, g_context.c_str(), #x);                     \
+        std::abort();                                                           \
+    }                                                                           \
+} while (0)
+
+#define REQUIRE_EQ(actual, expected) do {                                       \
+    if (!((actual) == (expected))) {                                            \
+        fprintf(stderr, "%s:%d: [%s] REQUIRE_EQ(%s, %s) failed\n  actual:   '%s'\n  expected: '%s'\n", \
+                __FILE__, __LINE__, g_context.c_str(), #actual, #expected,      \
+                std::string(actual).c_str(), std::string(expected).c_str());    \
+        std::abort();                                                           \
+    }                                                                           \
+} while (0)


not sure if we should reuse the testing.h framework here

Good fit I think, the continue-on-failure would show the whole table on a red run instead of the first abort. I would do it as a follow-up though, the assertions guard some control flow (parse before init) so it is a real rework of the test, not a swap.

ngxson · 2026-07-29T10:34:33Z

+common_http_client_ptr common_http_client_create(const std::string & url);
+
+// substitute the client creation, e.g. with a stub (ONLY for testing)
+void common_http_client_set_factory(common_http_client_ptr (*factory)(const std::string & url));


cosmetic: maybe typedef the callback into a shorter naming common_http_client_factory_fn

ngxson · 2026-07-29T10:35:45Z

+
+static const char * COMMIT = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";
+
+struct http_client_stub : common_http_client {


note that the same stub can be reused by other test cases in the future, we can make it more flexible if needed (TBD)

Yes, moving it to a shared test header is easy once a second user shows up

ServeurpersoCom · 2026-07-29T14:32:21Z

Remaining gpu-vulkan-intel-windows :

6
Start 26: test-model-resolution
25/33 Test #26: test-model-resolution .............***Exception: SegFault 0.24 sec

ngxson · 2026-07-29T14:44:26Z

+    common_http_client_set_factory([](const std::string & url) -> common_http_client_ptr {
+        return std::make_unique<http_client_stub>(url);
+    });
+


not 100% sure about this, but probably you need to define the factory as a static func, not a lambda?

A captureless lambda converts to a plain function pointer so both should be equivalent. But the latest commit makes the question moot for this test: the repos are now served by a real httplib server on the loopback and the test does not touch the factory anymore, so no C++ object crosses the module boundary at all, which is my best suspect for the Debug shared crash on the intel windows runner. The seam stays in common for future tests.

I will try replacing the stub by a mini httplib server on the loopback instead: for about the same line count the test becomes even more complete, the library exercises its real client and transport end to end, and no C++ object crosses the module boundary anymore, which is my best suspect for the Debug shared crash on the intel windows runner.

I see, yeah that makes sense. If the stub is defined at libcommon level, that would also solve the problem. but I'd agree that a loopback server works better in this case

ngxson · 2026-07-29T14:45:42Z

+    set_env("LLAMA_CACHE", cache_dir.string().c_str());
+
+    // an http endpoint keeps the client init from rejecting https on the
+    // builds without TLS support, the stub serves it either way
+    set_env("MODEL_ENDPOINT", "http://models.test/");


hmm, maybe also abstract out the set/get_env into common_get/set_env, so that it can be platform-independent similar to process.env in nodejs

Good idea, and it composes with the latest commit: the stub is gone (the repos are now served by a real httplib server on the loopback so nothing crosses the module boundary anymore), but the test still sets LLAMA_CACHE and MODEL_ENDPOINT, so I will promote the helper into common_get/set_env and use it there.

ServeurpersoCom · 2026-07-29T15:09:04Z

And now I can remove my http.cpp !

ServeurpersoCom · 2026-07-29T18:26:48Z

https://github.com/ServeurpersoCom/llama.cpp/actions/runs/30478466732

Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression.

Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3.

The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain.

WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards.

The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link.

Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites.

Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short.

Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering.

Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably.

The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably.

@ngxson

Address review from @ngxson

Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path.

@ngxson

Address review from @ngxson

The loopback server made the stub substitution unnecessary, the client init builds the real client directly again.

The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other.

ngxson · 2026-08-03T08:06:29Z

 }

-static std::pair<httplib::Client, common_http_url> common_http_client(const std::string & url) {
+class common_http_client {


this wrapper is unused now, maybe better to remove it?

(I mean revert to the non-ptr version on master)

@ngxson

address review from @ngxson

ngxson

nice, very clean now

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

@ngxson

…#26172) * tests: add model resolution test on synthetic repo listings Include download.cpp and arg.cpp inside a namespace with hf_cache monkey patched to serve hardcoded listings, so the resolution and the model handler assembly are tested end-to-end through the real CLI parsing, without modifying the tested code and without network access. Covers the primary, shard, mmproj, sidecar and preset resolution on layouts mimicking real vendor conventions, replays every case on permutations of the listing to assert determinism, and asserts the final wired paths for the spec type auto-selection, the -md precedence and the fallback suppression. * tests: keep model resolution checks active and let the handler resolve Replace assert with a REQUIRE macro alive in Release builds, key the fake hf_cache by repo id so the real handler init resolves every plan itself, check the exact shard sets, restrict the permutation exception to the order dependent picks, and cover dflash and eagle3. * tests: fix model resolution build on fatal warnings CI and Windows The namespaced copy of the sources leaves many static functions unused in this TU, exempt it from the unused warnings. Pre-include the windows headers so arg.cpp does not pull them inside the namespace. Declare the renamed copies of the download.h functions, verbatim from the header and renamed in sync by the macros, so missing declaration and missing prototype warnings are satisfied on every toolchain. * tests: fix winsock inclusion order for the model resolution test WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards. * tests: link cpp-httplib to the model resolution test The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link. * common_http_client * common: finish the http client wrapper Add the virtual Head, Get and Post methods and the passthrough setters to the common_http_client skeleton, move follow_location into the constructor, expose the underlying client for the ranged pull path, and rename the missed common_http_client_init call sites. * tests: rewrite model resolution on the http client stub Replace the namespace inclusion of the sources by a plain TU: the common_http_client factory returns a stub serving hardcoded HF API responses, so the real hf_cache parsing, resolution and CLI handler run against synthetic listings in an isolated cache directory. Failures print the named case, the reordering and the actual versus expected values, the assembly cases use the full command line as context, and the empty result cases are checked once to keep the logs short. * tests: fix the model resolution on Windows and the builds without TLS Assert the exact expected paths composed like the cache does instead of suffix matching on forward slashes, set the environment portably, and serve the stub through an http endpoint so the builds rejecting the https scheme still reach it. Pause the log so the negative cases can be replayed on every reordering. * tests: make the model resolution failures self explanatory Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably. * common: hold the http client factory behind exported functions The factory was an inline variable, and the Windows shared builds export functions but not data, so the executable and the DLL each had their own instance: the stub installed by the test was invisible to the library, which reached for the real endpoint and resolved nothing. Route the creation through functions compiled into the library and format the stub oids portably. * common: add the http client factory source missed in the previous commit * common: typedef the http client factory callback Address review from @ngxson * tests: serve the model resolution repos over the loopback Replace the client stub by a real httplib server bound to the loopback, so no C++ object crosses the module boundary anymore and the library exercises its own client and transport end to end. The debug shared build on Windows crashed inside the stubbed path. * common: add portable common_get_env and common_set_env helpers Address review from @ngxson * common: drop the http client factory left without a caller The loopback server made the stub substitution unnecessary, the client init builds the real client directly again. * common: read the model endpoint through the env helpers * nit: drop the stub leftovers from the model resolution test * common: align common_set_env and isolate the test cache per run The POSIX branch now behaves like _putenv_s, so the helper has a single contract on every platform, and common_get_env already reads an unset and an empty variable alike. The model resolution test keys its cache directory on the loopback port, where two concurrent runs on the same machine used to share one directory and the initial cleanup of either wiped the other. * tests: move the model resolution server into main * tests: support the DSpark sidecar resolution * common: revert the http client to the plain httplib client address review from @ngxson --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>

ServeurpersoCom requested a review from ggerganov as a code owner July 27, 2026 11:28

github-actions Bot added the testing Everything test related label Jul 27, 2026

ServeurpersoCom force-pushed the test-model-resolution branch from d22f91f to fc3d2da Compare July 28, 2026 16:14

ggerganov approved these changes Jul 28, 2026

View reviewed changes

ngxson reviewed Jul 28, 2026

View reviewed changes

ServeurpersoCom mentioned this pull request Jul 28, 2026

Refactor/extract model resolution #26247

Closed

ServeurpersoCom requested a review from a team as a code owner July 29, 2026 08:20

github-actions Bot added the examples label Jul 29, 2026

ngxson reviewed Jul 29, 2026

View reviewed changes

ngxson approved these changes Jul 29, 2026

View reviewed changes

ngxson reviewed Jul 29, 2026

View reviewed changes

ServeurpersoCom force-pushed the test-model-resolution branch from a915b7f to f6ac3ae Compare July 29, 2026 16:11

ServeurpersoCom and others added 7 commits August 2, 2026 19:27

tests: fix winsock inclusion order for the model resolution test

095b08b

WIN32_LEAN_AND_MEAN and winsock2.h before windows.h, so http.h does not redefine the socket types afterwards.

tests: link cpp-httplib to the model resolution test

81f1183

The test compiles its own copy of download.cpp, which calls httplib directly, and the private link of llama-common does not propagate the symbols under lld-link.

common_http_client

d2cb196

ServeurpersoCom added 14 commits August 2, 2026 19:27

tests: make the model resolution failures self explanatory

42e03e3

Resume the paused log before the failure report so the CI shows why the tested code bailed, and format the stub oids portably.

common: add the http client factory source missed in the previous commit

0e129e9

common: typedef the http client factory callback

3e247a7

Address review from @ngxson

common: add portable common_get_env and common_set_env helpers

22df0c2

Address review from @ngxson

common: drop the http client factory left without a caller

1818d5c

The loopback server made the stub substitution unnecessary, the client init builds the real client directly again.

common: read the model endpoint through the env helpers

c1058d3

nit: drop the stub leftovers from the model resolution test

d8e8575

tests: move the model resolution server into main

62dea50

tests: support the DSpark sidecar resolution

4f26fc6

ServeurpersoCom force-pushed the test-model-resolution branch from 2883c87 to 4f26fc6 Compare August 2, 2026 17:27

ngxson approved these changes Aug 3, 2026

View reviewed changes

ngxson reviewed Aug 3, 2026

View reviewed changes

common: revert the http client to the plain httplib client

bacf4a1

address review from @ngxson

ngxson approved these changes Aug 3, 2026

View reviewed changes

ggerganov approved these changes Aug 3, 2026

View reviewed changes

ServeurpersoCom merged commit ee0445c into ggml-org:master Aug 3, 2026
22 of 26 checks passed

rillomas mentioned this pull request Aug 4, 2026

Misc. bug: test-model-resolution crashes on Windows MSYS2 UCRT64 #26552

Closed


		static const char * COMMIT = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa";

		struct http_client_stub : common_http_client {

Uh oh!

Conversation

ServeurpersoCom commented Jul 27, 2026

Overview

Additional information

Requirements

Uh oh!

ServeurpersoCom commented Jul 28, 2026

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov commented Jul 28, 2026

Uh oh!

ServeurpersoCom commented Jul 28, 2026

Uh oh!

ServeurpersoCom commented Jul 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson commented Jul 28, 2026

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ServeurpersoCom commented Jul 29, 2026

Uh oh!

ServeurpersoCom commented Jul 29, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ServeurpersoCom commented Jul 28, 2026 •

edited

Loading

ServeurpersoCom Jul 28, 2026 •

edited

Loading