Add immediate isq predicates for qwen3

[EricLBuehler]

Summary by CodeRabbit

• New Features
  • Added informative logging to provide visibility into the application of immediate in-situ quantization (ISQ) during model loading. Users will now see messages indicating the ISQ type being applied and warnings if no ISQ predicates are detected.
• Bug Fixes
  • Standardized the handling of ISQ predicate methods across different model loaders to ensure consistent behavior.
• Improvements
  • Enhanced ISQ parsing to consider device context dynamically, improving compatibility across hardware types.
  • Updated device selection logic to better support CPU and Metal devices, respecting user preferences.
  • Deferred parsing of ISQ options until device context is available, improving configuration flexibility.

[coderabbitai]

## Walkthrough

The changes standardize method naming for immediate ISQ predicate retrieval in model loaders and add logging to provide visibility into ISQ predicate application during model loading. The `parse_isq_value` function signature is updated to accept an optional device parameter, enabling runtime device checks instead of compile-time flags. Command-line argument handling for ISQ options is modified to defer parsing until device context is available. Device initialization logic for Metal and CPU modes is improved. No control flow or logic is altered beyond these updates.

## Changes

| File(s)                                                                 | Change Summary                                                                                              |
|-------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|
| mistralrs-core/src/pipeline/loaders/normal_loaders.rs                   | Standardized method naming for ISQ predicate retrieval: replaced `immediate_isq_predicates_moqe` with `immediate_isq_predicates` in `Qwen3Loader`, and added `immediate_isq_predicates` to `Qwen3MoELoader`. |
| mistralrs-core/src/pipeline/normal.rs, mistralrs-core/src/pipeline/vision.rs | Added informational and warning logging around the application of immediate ISQ predicates during model loading. |
| mistralrs-core/src/pipeline/isq.rs                                      | Updated `parse_isq_value` to accept an optional device parameter and replaced compile-time Metal feature checks with runtime device checks. |
| mistralrs-bench/src/main.rs, mistralrs-server/src/main.rs, mistralrs-pyo3/src/lib.rs, mistralrs/examples/perplexity/main.rs, mistralrs-core/src/topology/mod.rs | Changed ISQ argument parsing to defer string-to-`IsqType` conversion until device context is available by modifying argument types and adding device-aware parsing calls. Improved Metal and CPU device initialization logic in server. |

## Poem

> A loader hops and logs its way,  
> ISQ predicates lead the play.  
> With names aligned and warnings bright,  
> Each model loads with clearer sight.  
> Rabbits cheer for tidy code,  
> And logs that light the ISQ road!  
> 🐇✨

✨ Finishing Touches

• 📝 Generate Docstrings

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Explain this complex logic.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
  • @coderabbitai explain this code block.
  • @coderabbitai modularize this function.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read src/utils.ts and explain its main purpose.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
  • @coderabbitai help me debug CodeRabbit configuration file.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

• @coderabbitai pause to pause the reviews on a PR.
• @coderabbitai resume to resume the paused reviews.
• @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
• @coderabbitai full review to do a full review from scratch and review all the files again.
• @coderabbitai summary to regenerate the summary of the PR.
• @coderabbitai generate docstrings to generate docstrings for this PR.
• @coderabbitai generate sequence diagram to generate a sequence diagram of the changes in this PR.
• @coderabbitai resolve resolve all the CodeRabbit review comments.
• @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
• @coderabbitai help to get help.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions]

Code Metrics Report

===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                3           62           53            0            9
 Dockerfile              1           41           22           10            9
 JSON                   12          107          106            0            1
 Makefile                1            6            5            0            1
 Python                 86         4045         3413          158          474
 Shell                   1           63           26           18           19
 Plain Text              3         3723            0         2413         1310
 TOML                   19          565          518            6           41
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       3            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               55         5012            0         3822         1190
 |- BASH                 8          104          101            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                22          757          634            1          122
 |- TOML                 2           75           63            0           12
 (Total)                           6081          919         3823         1339
-------------------------------------------------------------------------------
 Rust                  378       128798       114851         2703        11244
 |- Markdown           172         2205           29         1967          209
 (Total)                         131003       114880         4670        11453
===============================================================================
 Total                 564       142443       119013         9132        14298
===============================================================================

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (2)

mistralrs-core/src/pipeline/vision.rs (1)

412-415: Fix typo in warning message

There's a typo in the warning message: "deteced" should be "detected".

- warn!("No predicates for this model and ISQ setting deteced. ISQ will not be applied to any weights!");
+ warn!("No predicates for this model and ISQ setting detected. ISQ will not be applied to any weights!");

The added logging improves visibility into the ISQ predicate application process, which is helpful for debugging and understanding the model loading behavior.

🧰 Tools

🪛 GitHub Check: Typos

[warning] 414-414:
"deteced" should be "detected".

mistralrs-core/src/pipeline/normal.rs (1)

481-484: Fix typo in warning message

There's a typo in the warning message: "deteced" should be "detected".

- warn!("No predicates for this model and ISQ setting deteced. ISQ will not be applied to any weights!");
+ warn!("No predicates for this model and ISQ setting detected. ISQ will not be applied to any weights!");

The added logging improves visibility into the ISQ predicate application process, helping with debugging and understanding the model loading behavior.

🧰 Tools

🪛 GitHub Check: Typos

[warning] 483-483:
"deteced" should be "detected".

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 504401f and 24fc04c.

📒 Files selected for processing (3)

• mistralrs-core/src/pipeline/loaders/normal_loaders.rs (2 hunks)
• mistralrs-core/src/pipeline/normal.rs (1 hunks)
• mistralrs-core/src/pipeline/vision.rs (1 hunks)

🧰 Additional context used

🧬 Code Graph Analysis (1)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (2)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (15)

• immediate_isq_predicates (338-340)
• immediate_isq_predicates (514-516)
• immediate_isq_predicates (788-801)
• immediate_isq_predicates (1126-1139)
• immediate_isq_predicates (1388-1401)
• immediate_isq_predicates (1695-1697)
• immediate_isq_predicates (2025-2027)
• immediate_isq_predicates (2312-2341)
• immediate_isq_predicates (2621-2623)
• immediate_isq_predicates (2893-2895)
• immediate_isq_predicates (3216-3218)
• immediate_isq_predicates (3502-3515)
• immediate_isq_predicates (3828-3841)
• immediate_isq_predicates (4152-4196)
• config (78-78)

mistralrs-core/src/pipeline/isq.rs (1)

• immediate_isq_predicates (960-962)

🪛 GitHub Check: Typos

mistralrs-core/src/pipeline/vision.rs

[warning] 414-414:
"deteced" should be "detected".

mistralrs-core/src/pipeline/normal.rs

[warning] 483-483:
"deteced" should be "detected".

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Clippy
• GitHub Check: Docs
• GitHub Check: Check (windows-latest, stable)
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)
• GitHub Check: Check (ubuntu-latest, stable)
• GitHub Check: Test Suite (ubuntu-latest, stable)

🔇 Additional comments (2)

mistralrs-core/src/pipeline/loaders/normal_loaders.rs (2)

3007-3009: Looks good - adds the standard immediate_isq_predicates method for Qwen3Loader.

This change standardizes the method naming convention by implementing the standard immediate_isq_predicates method for the Qwen3Loader, delegating to isq_layer_regexes. This follows the same pattern used by other model loaders throughout the codebase.

---

3192-3194: Looks good - adds the standard immediate_isq_predicates method for Qwen3MoELoader.

This change implements the standard immediate_isq_predicates method for the Qwen3MoELoader, delegating to isq_layer_regexes, while preserving the existing immediate_isq_predicates_moqe method (line 3195). This standardization ensures consistency across all model loaders.

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (2)

mistralrs-pyo3/src/lib.rs (1)

1419-1419: Consider passing device context from runner

In the send_re_isq method, parse_isq_value is called with None for the device parameter. While this works, it might be more consistent to retrieve and pass the actual device context from the runner to ensure proper device-specific ISQ type selection.

-let request = _Request::ReIsq(parse_isq_value(&dtype, None)?);
+let request = _Request::ReIsq(parse_isq_value(&dtype, Some(self.runner.device()))?);

However, this would require ensuring the runner.device() method exists or implementing it if it doesn't. The current approach is valid if the default non-Metal behavior is desired in this context.

mistralrs-server/src/main.rs (1)

302-308: Side-effect inside let device reduces readability

Mutating args.no_paged_attn inside the ternary expression hides state changes:

let device = if args.cpu {
    args.no_paged_attn = true; // side-effect
    Device::Cpu
} else {
    Device::new_metal(0)?
};

Consider setting args.no_paged_attn in a separate, explicit block before/after device selection to make intent clearer and avoid accidental future regressions.

📜 Review details

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 24fc04c and 107089f.

📒 Files selected for processing (8)

• mistralrs-bench/src/main.rs (3 hunks)
• mistralrs-core/src/pipeline/isq.rs (1 hunks)
• mistralrs-core/src/pipeline/normal.rs (1 hunks)
• mistralrs-core/src/pipeline/vision.rs (1 hunks)
• mistralrs-core/src/topology/mod.rs (1 hunks)
• mistralrs-pyo3/src/lib.rs (2 hunks)
• mistralrs-server/src/main.rs (5 hunks)
• mistralrs/examples/perplexity/main.rs (1 hunks)

✅ Files skipped from review due to trivial changes (1)

• mistralrs-core/src/pipeline/vision.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• mistralrs-core/src/pipeline/normal.rs

🧰 Additional context used

🧬 Code Graph Analysis (4)

mistralrs/examples/perplexity/main.rs (1)

mistralrs-core/src/pipeline/isq.rs (1)

• parse_isq_value (62-125)

mistralrs-core/src/topology/mod.rs (1)

mistralrs-core/src/pipeline/isq.rs (1)

• parse_isq_value (62-125)

mistralrs-server/src/main.rs (1)

mistralrs-core/src/pipeline/isq.rs (1)

• parse_isq_value (62-125)

mistralrs-core/src/pipeline/isq.rs (15)

mistralrs-core/src/pipeline/vision.rs (1)

• device (813-815)

mistralrs-core/src/pipeline/normal.rs (1)

• device (961-963)

mistralrs-core/src/pipeline/mod.rs (1)

• device (151-151)

mistralrs-core/src/models/gemma.rs (1)

• device (628-630)

mistralrs-core/src/models/llama.rs (1)

• device (647-649)

mistralrs-core/src/models/gemma2.rs (1)

• device (687-689)

mistralrs-core/src/models/mistral.rs (1)

• device (669-671)

mistralrs-core/src/models/mixtral.rs (1)

• device (759-761)

mistralrs-core/src/models/phi3.rs (1)

• device (698-700)

mistralrs-core/src/models/qwen2.rs (1)

• device (643-645)

mistralrs-core/src/models/phi2.rs (1)

• device (694-696)

mistralrs-core/src/models/starcoder2.rs (1)

• device (686-688)

mistralrs-core/src/pipeline/loaders/vision_loaders.rs (1)

• device (74-74)

mistralrs-core/src/xlora_models/gemma.rs (1)

• device (833-835)

mistralrs-core/src/vision_models/mllama/mod.rs (1)

• device (187-189)

⏰ Context from checks skipped due to timeout of 90000ms (8)

• GitHub Check: Docs
• GitHub Check: Clippy
• GitHub Check: Check (macOS-latest, stable)
• GitHub Check: Check (windows-latest, stable)
• GitHub Check: Check (ubuntu-latest, stable)
• GitHub Check: Test Suite (windows-latest, stable)
• GitHub Check: Test Suite (ubuntu-latest, stable)
• GitHub Check: Test Suite (macOS-latest, stable)

🔇 Additional comments (6)

mistralrs/examples/perplexity/main.rs (1)

77-77: Updated to pass device context to parse_isq_value

The function call has been modified to accommodate the new signature of parse_isq_value, passing None as the device parameter. This change aligns with the broader refactoring that enables runtime device-aware ISQ type selection instead of relying on compile-time feature flags.

mistralrs-core/src/topology/mod.rs (1)

110-110: Updated to pass device context to parse_isq_value

The function call has been modified to accommodate the new signature of parse_isq_value, passing None as the device parameter. This allows for ISQ type selection at runtime based on device characteristics rather than compile-time feature flags.

mistralrs-pyo3/src/lib.rs (1)

685-685: Properly passing device context to parse_isq_value

The function call now passes the actual device context (Some(device)) to parse_isq_value. This is appropriate since in the Runner::new constructor, the device is already available, allowing for device-aware ISQ type selection.

mistralrs-core/src/pipeline/isq.rs (2)

62-64: Enhanced function signature for runtime device awareness

The parse_isq_value function now accepts an optional device context parameter, allowing for runtime determination of ISQ types based on device characteristics rather than compile-time feature flags. This is a good improvement for flexibility and code maintainability.

---

65-75: Improved ISQ type selection logic for Metal devices

The matching logic now uses runtime conditions based on whether the device is a Metal device, replacing what was likely compile-time feature flags. This allows for more flexible deployment across different hardware and better adapts to the specific runtime environment.

For numeric string inputs ("2", "3", "4", "6", "8"), Metal devices use AFQ variants while non-Metal devices use QK variants or Q8_0. This maintains backward compatibility while adding device-specific optimization.

mistralrs-bench/src/main.rs (1)

505-506: After propagating errors, loader call stays unchanged

Once isq is an Option<IsqType> (see previous comment) the call here remains correct – no additional changes required.

[coderabbitai]

🛠️ Refactor suggestion

Early-stage CLI validation for --isq was lost – re-introduce via value_parser

By changing the field to Option<String> without a value_parser, invalid ISQ values will now silently pass clap’s parsing stage and only be caught (maybe) much later in the program flow. You can keep the defer-to-device behaviour while still validating user input early:

-#[arg(long = "isq")]
-in_situ_quant: Option<String>,
+#[arg(long = "isq", value_parser = |s: &str| parse_isq_value(s, None))]
+in_situ_quant: Option<IsqType>,

This preserves the new string syntax at the CLI, catches typos immediately, and still allows you to decide later whether the chosen ISQ is legal for the detected device.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	#[arg(long = "isq")]
	in_situ_quant: Option<String>,
	#[arg(long = "isq", value_parser = |s: &str| parse_isq_value(s, None))]
	in_situ_quant: Option<IsqType>,

🤖 Prompt for AI Agents

In mistralrs-bench/src/main.rs around lines 303 to 305, the CLI argument for
--isq was changed to Option<String> without a value_parser, causing loss of
early validation and allowing invalid values to pass silently. To fix this,
reintroduce a value_parser for the in_situ_quant field that validates the input
string against allowed ISQ values during CLI parsing. This will catch invalid
inputs immediately while preserving the option to defer detailed validation to
later in the program.

[coderabbitai]

⚠️ Potential issue

Parsing errors are swallowed – user never knows their ISQ argument was ignored

ok() converts the Result into an Option, discarding the error. If the user types an invalid value (e.g. --isq q9k) the model will just load un-quantised, giving confusing behaviour.

-let isq = args
-    .in_situ_quant
-    .as_ref()
-    .and_then(|isq| parse_isq_value(isq, Some(&device)).ok());
+let isq = match &args.in_situ_quant {
+    Some(s) => Some(parse_isq_value(s, Some(&device))?),
+    None    => None,
+};

Now an invalid flag stops execution with a clear error instead of silently degrading.

🤖 Prompt for AI Agents

In mistralrs-bench/src/main.rs around lines 493 to 497, the code currently uses
ok() to convert the Result from parse_isq_value into an Option, which discards
any parsing errors and silently ignores invalid ISQ arguments. To fix this,
change the code to properly handle the Result by checking for errors and
returning or displaying a clear error message to the user if parsing fails,
preventing silent degradation and stopping execution on invalid input.

[coderabbitai]

🛠️ Refactor suggestion

Re-add clap-level validation for --isq

Same concern as in the bench binary: turning the field into Option<String> removes early validation. Add a value_parser or custom ArgEnum so that typos are rejected before the (potentially expensive) model load begins.

🤖 Prompt for AI Agents

In mistralrs-server/src/main.rs around lines 122 to 123, the in_situ_quant field
is defined as Option<String> without clap-level validation, which allows typos
to pass through. To fix this, replace Option<String> with a type that uses
clap's value_parser or a custom ArgEnum to enforce valid values at argument
parsing time, ensuring invalid inputs are rejected early before model loading.

[coderabbitai]

⚠️ Potential issue

Silent drop of invalid --isq value – propagate the error

Identical issue to the bench tool: using .ok() discards the error.

-let isq = args
-    .in_situ_quant
-    .as_ref()
-    .and_then(|isq| parse_isq_value(isq, Some(&device)).ok());
+let isq = match &args.in_situ_quant {
+    Some(s) => Some(parse_isq_value(s, Some(&device))?),
+    None    => None,
+};

This will return a 500 at startup with a clear error instead of quietly ignoring the flag, which is preferable for an operator-facing server binary.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	let isq = args
	.in_situ_quant
	.as_ref()
	.and_then(|isq| parse_isq_value(isq, Some(&device)).ok());
	let isq = match &args.in_situ_quant {
	Some(s) => Some(parse_isq_value(s, Some(&device))?),
	None => None,
	};

🤖 Prompt for AI Agents

In mistralrs-server/src/main.rs around lines 434 to 438, the code silently drops
errors from parsing the `--isq` flag by using `.ok()`, which discards the error.
Modify the code to propagate the error instead of ignoring it, so that invalid
`--isq` values cause a clear startup failure with an appropriate error message,
preventing silent misconfiguration.