feat: v0.3 agent MCP tools — plan/execute/diagnose/compare/model-info - #245
Conversation
…y tools - Mark v0.1.0 and v0.2 as shipped, remove "current" and "next" labels - Restructure v0.3 section around 5 new MCP agent autonomy tools (`execute_and_observe`, `plan_optimization`, `diagnose_and_fix`, `compare_results`, `get_model_info`) - Rename "Backlog / v0.3+" to clearer v0.4 and Backlog sections with specific features - Remove graduation criteria sections (Tauri desktop, MultiLoRA) for conciseness - Consolidate v0.2 shipped work (persistent MCP, component splits, security headers, agent platform phases) - Move experimental features to backlog with blocker notes (MultiLoRA, cloud sync, WebGPU) - Update CI pipeline description to include test tiers and pytest - Simplify status legend table formatting
Coordinated version bump from olive-ai 0.12.1 to 0.13.0:
• 8 new passes added to TS catalog, MCP knowledge base, and recipe builder
(MobiusBuilder, QairtPipeline, KQuant, OnnxKquantQuantization,
QuantizeEmbeddingInt8, ShareEmbeddingLmHead, SimplifiedLayerNormToRMSNorm,
OnnxDiscrepancyCheck)
• KQuant added as new quantization method ("kquant" in type union + allowlists)
• trust_remote_code default flip handled in recipe builder + pipeline advisory
• QNN ABI EP added to hardware profiles and provider conflicts
• CROSS_PASS_RULES for QairtPipeline and SimplifiedLayerNormToRMSNorm EP constraints
• Removed-pass warning rule for MobiusModelBuilder/QairtPreparation/QairtGenAIBuilder
• Migration module (src/lib/passMigration.ts) with pass name renames, removals,
and parameter migration infrastructure
• Migration integrated into pipelineStore replaceState + rehydration paths
• Venv spec bumped to v5 (olive-ai>=0.12.0,<1)
• Sync script version guard for 0.13.x
Tests: 1052 unit tests passing, 5 property-based tests (idempotence, validity,
preservation, exclusion, recipe schema), 6 integration test fixtures.
Lint: 0 errors. Recipe validation: passes.
Register execute_and_observe, plan_optimization, diagnose_and_fix, compare_results, and get_model_info in ALLOWED_MCP_TOOL_NAMES under a Phase 3 comment group. Requirements: 2.2, 4.2, 6.2, 8.2, 10.2
Add 5 new agent autonomous loop tools to _TOOL_IMPORTS in mcp_server.py and ALLOWED_MCP_TOOL_NAMES in allowedTools.ts: - execute_and_observe → agent_execute - plan_optimization → agent_planner - diagnose_and_fix → agent_diagnosis - compare_results → agent_compare - get_model_info → agent_model_info Part of v0.3-agent-mcp-tools (tasks 1.1 + 1.2).
Implement the get_model_info MCP tool that provides HuggingFace model metadata lookup with heuristic fallback for autonomous agent planning. - Validates model_id (1-256 chars) - Attempts HF API call (urllib, 3s timeout) for exact param count - Falls back to inferParamBillions regex heuristics (ported from src/lib/vramEstimate.ts) on any HF failure - Computes VRAM estimate (params_b * 2 FP16 baseline) - Classifies model_type via _normalize_model_type from strategy_advisor - Recommends int4 (>=6B) or int8 (<6B) quantization - Confidence: high (HF API), medium (explicit size token), low (family default) - Zero new pip dependencies, no module-level network I/O - Wrapped in top-level try/except for internal_error safety Implements: Requirements 9.1-9.8, 11.1, 11.3, 12.1, 12.5
Implements the diagnose_and_fix MCP tool (agent_diagnosis.py) for the Phase 3 autonomous agent loop. - Validates error_message (1-4000 chars) and recipe (must be dict) - Calls troubleshoot_olive_error internally with config context - Applies RFC 7386 JSON Merge Patch when KB entry has updated_config - Generates human-readable change descriptions - Best-effort recipe validation through Studio bridge - Maps fix_confidence: high/medium/low/none based on KB match quality - Top-level try/except for internal_error safety net - Zero new pip dependencies (stdlib only + internal imports) Requirements: 5.1-5.9, 6.3, 11.1, 11.3, 12.1-12.7
- Use a dedicated discrepancy test-data directory - Classify QNN ABI as NPU - Classify KQuant as PyTorch-native
Implements the execute_and_observe MCP tool for the Phase 3 autonomous agent loop. Submits a recipe to Olive Studio via the loopback bridge, polls job status at 2-second intervals until a terminal state is reached or the effective timeout expires. Key behaviors: - Timeout clamping: min(max(timeout or 600, 10), 1800) - Terminal states: completed, failed, cancelled - Terminal-at-timeout-boundary: terminal wins (timed_out: false) - Pre-submission errors: no side_effect field - Post-submission results: side_effect: True - Logs capped at 200 entries, artifact refs as basenames only - Top-level try/except for internal_error safety Requirements: 1.1-1.12, 2.3, 2.4, 2.5, 11.1, 11.3, 12.1-12.7
Implements the compare_results MCP tool (task 6.1) for multi-job
comparison with preference-weighted scoring.
- Validates job_ids count (2-10) and format (^[A-Za-z0-9_-]{1,128}$)
- Normalizes preference (latency/size/accuracy/balanced)
- Fetches job status via studio_request loopback bridge
- Excludes non-terminal, failed, or unfetchable jobs
- Min-max normalizes metrics with lower-is-better inversion
- Applies 2x weight for preferred metric, 1x for others
- Selects highest scored job as winner
- Returns structured comparison with side_effect: False
- Top-level try/except for internal_error safety
- Zero new pip dependencies (stdlib + studio_loopback only)
Requirements: 7.1-7.8, 8.3, 11.1, 11.3, 12.1-12.7
Implements the plan_optimization MCP tool that converts natural-language optimization intents into UIState patches for the Olive Studio frontend. - Parses intent via regex/keyword dispatch for hardware target, model reference, and optimization goal - Calls existing get_quantization_strategy() and get_hardware_optimization_guide() internally - Composes UIState patch with ihvProvider, cudaVersion, passes, etc. - Supports hardware_probe override and model_id for model-type inference - Validates patch via Studio bridge (graceful degradation if unavailable) - Generates 0-3 alternative approaches - Returns structured response with reasoning and validation status - Zero new pip dependencies; uses only Python stdlib + internal modules - No module-level network calls (lazy-import safe) Requirements: 3.1-3.9, 4.3, 11.1, 11.3, 12.1-12.7
Add property-based test verifying that plan_optimization returns 'unparseable_intent' error for any input string containing none of the recognized hardware keywords, model reference patterns, or optimization goal keywords. Uses hypothesis with 100 examples, generating safe text from a filtered alphabet that excludes all trigger words. Mocks studio_request to prevent network calls. Validates: Requirements 3.5 Feature: v0.3-agent-mcp-tools, Property 3: Unparseable Intent Rejection
- Map planner hardware targets to execution providers - Use the MCP proxy toolName field
- Avoid serializing scripts as discrepancy data directories - Add defaults for required pass flags - Require explicit remote-code trust
Property 7: VRAM Estimate Arithmetic - verifies estimated_vram_gb == params_b * 2.0 Property 8: Recommended Quantization Threshold - int4 for >= 6.0, int8 for < 6.0 Both use hypothesis with max_examples=100 and mock _fetch_hf_metadata. Adds hypothesis>=6.100.0 to dev dependencies. Validates: Requirements 9.3, 9.5
|
CodeFactor found multiple issues: Complex Method
|
# Conflicts: # olive-mcp-server/olive_mcp_server/knowledge_base/compatibility_matrix.json # olive-mcp-server/olive_mcp_server/knowledge_base/hardware_profiles.json # olive-mcp-server/olive_mcp_server/knowledge_base/troubleshooting.json # scripts/sync-pass-catalog.mjs # src/lib/__tests__/passMigrationIntegration.test.ts # src/lib/__tests__/passMigrationPBT.test.ts # src/lib/__tests__/passParameterValidation.test.ts # src/lib/__tests__/pipelineValidation.test.ts # src/lib/oliveRecipeBuilder.ts # src/lib/passCatalog.test.ts # src/lib/passMigration.ts # src/lib/pipelineValidation.ts # src/lib/qnnReadiness.test.ts # src/lib/stores/pipelineStore.ts # src/lib/venvFamily.ts # src/server/services/venv/spec.test.ts # src/server/services/venv/spec.ts # src/types.ts
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 26
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@olive-mcp-server/olive_mcp_server/tools/agent_compare.py`:
- Around line 205-208: Update the job ID validation in compare_results to reject
duplicate IDs after each ID passes the existing format check. Track previously
seen IDs and return the established invalid_job_id error for any repeated value,
before creating scoreable entries or proceeding to comparison.
- Around line 200-208: Update the validation error responses in agent_compare.py
lines 200-208 to include side_effect: False, and update the internal-error
response in agent_compare.py lines 286-288 likewise. Ensure every response path
in the relevant function, including invalid job counts, invalid job IDs, and
unexpected errors, conforms to the documented schema.
- Around line 120-143: Update the score calculation around the metric loop and
final score so every compared job uses the same selected metric set, requiring
the preferred metric when one is configured. Do not let missing metrics reduce
total_weight; instead apply the established missing-metric penalty consistently
before ranking, or reject jobs lacking the required preference metric. Preserve
metric direction inversion and weighting for available values.
- Around line 235-240: Update the comparison flow around _extract_metrics in
olive-mcp-server/olive_mcp_server/tools/agent_compare.py:235-240 to obtain
latency, model size, and accuracy from the Studio result endpoint or artifact
schema that owns those optimization scores, rather than reading latestMetrics
GPU telemetry; retain no_comparable_metrics only when no optimization score
exists. Update the corresponding status fixtures in
olive-mcp-server/tests/test_agent_compare.py:26-40 and
olive-mcp-server/tests/test_agent_compare_props.py:59-67 to represent the real
result contract, with no direct reliance on latestMetrics for optimization
scores.
In `@olive-mcp-server/olive_mcp_server/tools/agent_diagnosis.py`:
- Around line 79-90: Update _describe_changes and the changes_made construction
in olive-mcp-server/olive_mcp_server/tools/agent_diagnosis.py (lines 79-90) to
compare the original recipe with fixed_recipe, reporting effective additions,
removals, replacements, and nested changes rather than relying only on
updated_config; ensure empty-object patches replacing scalars or creating
missing keys are described. Extend
olive-mcp-server/tests/test_agent_diagnosis.py (lines 291-323) with cases
covering removal of an absent key and empty-object replacement or creation,
asserting the resulting changes_made descriptions.
In `@olive-mcp-server/olive_mcp_server/tools/agent_execute.py`:
- Around line 114-119: The malformed success-shaped submission response must be
treated as an uncertain side effect, not safely retryable. In
olive-mcp-server/olive_mcp_server/tools/agent_execute.py:114-119, update the
missing-job-ID error returned by the submission flow to include side_effect:
True. In olive-mcp-server/tests/test_agent_execute_props.py:199-223, update the
property expectation for this post-submission failure to require side_effect:
True.
- Around line 163-166: Update the metrics handling in the agent execution flow
so the result JSON preserves scoreable optimization metrics such as latency_ms,
model_size_mb, and accuracy instead of assigning latestMetrics/GpuMetrics
directly to last_metrics. Store GPU telemetry separately when needed, and ensure
compare_results receives the scoreable result metrics.
In `@olive-mcp-server/olive_mcp_server/tools/agent_model_info.py`:
- Line 103: Update the model-size classification logic around the shared
(“mistral”, “mixtral”) fallback so Mixtral MoE identifiers are handled by a
dedicated heuristic before that generic fallback, rather than being classified
as 7B; ensure Mixtral-8x7B yields an appropriate MoE parameter/VRAM estimate.
Add a fallback test covering “mistralai/Mixtral-8x7B-Instruct-v0.1”.
- Around line 261-264: Update the model-type computation in the metadata-derived
fields to pass the returned architecture into _normalize_model_type, allowing
opaque model IDs to classify correctly from values such as
WhisperForConditionalGeneration. Add a test covering an opaque model ID with a
Whisper architecture and assert the resulting model type.
In `@olive-mcp-server/olive_mcp_server/tools/agent_planner.py`:
- Around line 67-76: Remove the unused _HARDWARE_TO_PROVIDER mapping from
agent_planner.py. Keep _infer_provider and _normalize_provider as the active
provider-resolution paths, ensuring canonical execution-provider IDs remain the
single source of truth for ihvProvider.
- Around line 261-276: Update the precision-flip alternatives in the planner
logic around the int4/int8 branches so their ui_state_patch does not retain the
primary strategy["pass_chain"], which may encode the original precision-specific
AWQ path. Remove the passChain assignment from both alternatives and let Studio
derive a chain matching the requested quantPrecision.
- Around line 364-379: Update the input-validation block in the planner entry
point to require model_id, when provided, to be a string before applying
len(model_id), and require hardware_probe, when provided, to be a mapping/object
with the expected structure; return err("invalid_input", ...) for invalid
shapes. In _compose_ui_state_patch, validate or constrain copied cudaVersion and
openvinoTargetDevice values to scalar string values before placing them in the
UIState patch, preserving the invalid_input contract.
- Line 16: Replace the cross-module import of private _normalize_model_type in
agent_planner with a public normalize_model_type function exposed by
strategy_advisor. Add the public wrapper in strategy_advisor, preserving the
existing _normalize_model_type behavior, and update agent_planner references to
use the public symbol.
- Around line 407-415: Handle error dictionaries returned by
get_quantization_strategy before passing strategy to _compose_ui_state_patch. In
the planner method containing the shown strategy and guide logic, detect
strategy["error"], return the plan_optimization tool’s established
unsupported-hardware error response with the underlying error as detail, and
verify the error code matches that tool’s contract.
- Around line 118-137: Update _infer_provider to detect CPU-only hardware
intents and return CPUExecutionProvider before the existing OpenVINO/intel
branch. Ensure the default "Intel Core i9 CPU" intent resolves to
CPUExecutionProvider while preserving OpenVINO handling for non-CPU Intel
targets.
- Around line 32-44: Update plan_optimization to route structured and explicit
OpenVINO hardware intents through parse_hardware_target before selecting the
quantization strategy, rather than relying on the order-sensitive
_HARDWARE_KEYWORDS matches. Ensure inputs such as “intel npu” and “openvino npu”
resolve to the canonical Intel Core Ultra NPU (OpenVINO) target and preserve the
resulting openvinoTargetDevice: NPU patch for get_quantization_strategy.
- Around line 140-153: Update _normalize_provider to return only recognized
canonical execution-provider IDs from the canonical mapping; replace the
unknown-value passthrough with None so unrecognized hardware_probe strings are
dropped before reaching the ihvProvider patch.
- Around line 319-336: Update _validate_patch to inspect the
validate_ui_state_recipe success payload rather than only response["error"]: use
the projected is_runnable value and return validated=False with an appropriate
note when it is false, including schema_errors or isBlocked details when
present. Preserve the Studio-unavailable handling and successful path for
runnable recipes, and update _mock_studio_request_success to emit the same
ok/isRunnable-or-is_runnable/schema-errors bridge shape.
In `@olive-mcp-server/tests/test_agent_compare_props.py`:
- Around line 115-155: Strengthen test_balanced_gives_equal_weights and
test_specific_preference_increases_metric_contribution by independently
calculating expected weighted scores and asserting each result matches, rather
than only checking the 0–1 range. Include generated sparse metric inputs to
verify the scorer’s missing-metric policy, and compare balanced, latency, size,
and accuracy outputs against the corresponding explicit weight calculations.
In `@olive-mcp-server/tests/test_agent_error_structure.py`:
- Around line 187-214: Remove the optional-import handling and skipif markers
around plan_optimization in _PLANNER_AVAILABLE and _PLANNER_ERROR_CASES. Import
the required planner tool directly so failures to import agent_planner cause the
test suite to fail rather than skipping its cases.
- Around line 272-284: Update the _mock_request comparison responses for job-a
and job-b to place latency, model-size, and accuracy under the status endpoint’s
actual result field rather than latestMetrics. Preserve latestMetrics only as
telemetry where applicable, and ensure the compare_results test scores using the
actual status-response result field.
In `@olive-mcp-server/tests/test_agent_planner_props.py`:
- Around line 26-49: Replace the duplicated trigger lists and _ALL_TRIGGERS
filtering source in the relevant test setup with the exported planner constants
from agent_planner, including _HARDWARE_KEYWORDS, _OPTIMIZATION_KEYWORDS, and
the planner’s model-pattern definitions such as _MODEL_PATTERNS. Remove or stop
using the unreachable _HF_MODEL_PATTERN check so generated strings are filtered
using the same recognition rules as the planner; retain literal lists only in a
separate agreement/regression test if needed.
In `@olive-mcp-server/tests/test_agent_planner.py`:
- Around line 147-158: Add a test alongside the existing provider normalization
coverage in the planner test suite that passes hardware_probe with ihvProvider
set to "not-a-provider", mocks studio_request consistently, and asserts the
optimization succeeds while ui_state_patch["ihvProvider"] falls back to
"CUDAExecutionProvider" rather than preserving the unknown label.
- Around line 418-448: Add boundary and wrong-type validation tests around
plan_optimization: verify intent of exactly 2000 characters and model_id of
exactly 200 characters are accepted, and parameterize non-string intent,
non-dict hardware_probe, and non-string model_id inputs to assert invalid_input
rather than internal_error. Reuse the existing studio_request monkeypatch and
run the tests with python -m pytest tests -q from the olive-mcp-server
directory.
- Around line 60-70: Update the provider assertion in the test_agent_planner
test to require the exact expected ihvProvider value, CUDAExecutionProvider, for
the NVIDIA int4 quantization intent. Remove the alternative TensorRT acceptance
while preserving the existing validation and quantization assertions.
- Around line 106-113: Update the CNN/OpenVINO assertions in plan_optimization
to verify the exact OpenVINOExecutionProvider value, enabled INT8 static
quantization settings, and the CNN/static pass-chain configuration. Add a
CNN-specific reasoning or key assertion such as model_type=cnn or pruning,
replacing the current loose provider substring OR check.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 068385c4-b435-4244-93f0-19879c15de2e
📒 Files selected for processing (22)
olive-mcp-server/olive_mcp_server/mcp_server.pyolive-mcp-server/olive_mcp_server/tools/__init__.pyolive-mcp-server/olive_mcp_server/tools/agent_compare.pyolive-mcp-server/olive_mcp_server/tools/agent_diagnosis.pyolive-mcp-server/olive_mcp_server/tools/agent_execute.pyolive-mcp-server/olive_mcp_server/tools/agent_model_info.pyolive-mcp-server/olive_mcp_server/tools/agent_planner.pyolive-mcp-server/pyproject.tomlolive-mcp-server/tests/test_agent_compare.pyolive-mcp-server/tests/test_agent_compare_props.pyolive-mcp-server/tests/test_agent_diagnosis.pyolive-mcp-server/tests/test_agent_diagnosis_props.pyolive-mcp-server/tests/test_agent_error_structure.pyolive-mcp-server/tests/test_agent_execute.pyolive-mcp-server/tests/test_agent_execute_props.pyolive-mcp-server/tests/test_agent_lazy_import.pyolive-mcp-server/tests/test_agent_model_info.pyolive-mcp-server/tests/test_agent_model_info_props.pyolive-mcp-server/tests/test_agent_planner.pyolive-mcp-server/tests/test_agent_planner_props.pyolive-mcp-server/tests/test_call_tool.pysrc/server/services/mcp/allowedTools.ts
🔗 Linked repositories identified
CodeRabbit considers these linked repositories for cross-repo context during reviews:
tonythethompson/QuickShell(manual)tonythethompson/numan(manual)tonythethompson/dependency-chain-substrate(manual)
📜 Review details
⏰ Context from checks skipped due to timeout. (2)
- GitHub Check: Greptile Review
- GitHub Check: python-tests
🧰 Additional context used
📓 Path-based instructions (11)
olive-mcp-server/**/*.{py,toml,txt}
📄 CodeRabbit inference engine (CLAUDE.md)
Pin the Python
mcpdependency to<2, because version 2.x removesmcp.server.fastmcpand breaks imports.
Files:
olive-mcp-server/pyproject.tomlolive-mcp-server/tests/test_agent_lazy_import.pyolive-mcp-server/tests/test_agent_planner_props.pyolive-mcp-server/olive_mcp_server/tools/__init__.pyolive-mcp-server/tests/test_agent_diagnosis_props.pyolive-mcp-server/olive_mcp_server/mcp_server.pyolive-mcp-server/tests/test_call_tool.pyolive-mcp-server/olive_mcp_server/tools/agent_model_info.pyolive-mcp-server/tests/test_agent_compare.pyolive-mcp-server/tests/test_agent_model_info_props.pyolive-mcp-server/olive_mcp_server/tools/agent_planner.pyolive-mcp-server/tests/test_agent_error_structure.pyolive-mcp-server/olive_mcp_server/tools/agent_execute.pyolive-mcp-server/olive_mcp_server/tools/agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute_props.pyolive-mcp-server/olive_mcp_server/tools/agent_compare.pyolive-mcp-server/tests/test_agent_compare_props.pyolive-mcp-server/tests/test_agent_planner.pyolive-mcp-server/tests/test_agent_model_info.pyolive-mcp-server/tests/test_agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute.py
**/*
📄 CodeRabbit inference engine (CLAUDE.md)
Do not run real Olive jobs in CI or VMs because they download models and CUDA wheels.
**/*: Do not run real Olive GPU workloads or model downloads in CI; use mocks or CPU-only flows.
When changing the threat model or fixing critical findings, update the review snapshot and document the local-trust model in user-facing documentation.
Files:
olive-mcp-server/pyproject.tomlsrc/server/services/mcp/allowedTools.tsolive-mcp-server/tests/test_agent_lazy_import.pyolive-mcp-server/tests/test_agent_planner_props.pyolive-mcp-server/olive_mcp_server/tools/__init__.pyolive-mcp-server/tests/test_agent_diagnosis_props.pyolive-mcp-server/olive_mcp_server/mcp_server.pyolive-mcp-server/tests/test_call_tool.pyolive-mcp-server/olive_mcp_server/tools/agent_model_info.pyolive-mcp-server/tests/test_agent_compare.pyolive-mcp-server/tests/test_agent_model_info_props.pyolive-mcp-server/olive_mcp_server/tools/agent_planner.pyolive-mcp-server/tests/test_agent_error_structure.pyolive-mcp-server/olive_mcp_server/tools/agent_execute.pyolive-mcp-server/olive_mcp_server/tools/agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute_props.pyolive-mcp-server/olive_mcp_server/tools/agent_compare.pyolive-mcp-server/tests/test_agent_compare_props.pyolive-mcp-server/tests/test_agent_planner.pyolive-mcp-server/tests/test_agent_model_info.pyolive-mcp-server/tests/test_agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute.py
olive-mcp-server/pyproject.toml
📄 CodeRabbit inference engine (REVIEW.md)
Constrain the Python
mcpdependency to<2, matching the supported FastMCP import path and CI configuration.
Files:
olive-mcp-server/pyproject.toml
src/**/*.{ts,tsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
src/**/*.{ts,tsx}: Match existing naming, file layout, and TypeScript patterns insrc/.
Put shared recipe logic insrc/lib/, especiallypipelineValidation.ts,oliveRecipeBuilder.ts, andrecipePipeline.ts.
src/**/*.{ts,tsx}: Route every UI state mutation throughcommitUiStateUpdateinsrc/lib/pipelineValidation.tsso invariants are enforced; usereplaceStatefor recipe imports and preset loads.
UseusePipelineState()as the shorthand hook for reading pipeline state.
Avoidexport *barrel imports; import directly from the actual module file.Follow the React conventions in
docs/REACT_BEST_PRACTICES.md, especially eliminating waterfalls, avoiding barrel imports, and deferring non-critical third-party libraries.
src/**/*.{ts,tsx}: Keep pipeline and recipe validation logic in shared libraries rather than duplicating it in IHV cell helpers or inspectors.
Keep the UI AI provider catalog synchronized with the server registry, preferably through a shared provider ID list or a synchronization test; register new providers in both places.
Add coverage forrecipe-graph/and under-tested libraries includingpassCatalog,oliveRecipeHub,jobHistoryStore, andvramEstimate.
Confirm whetherEnterpriseInfraPanelandPerformanceMetricsare required; remove them if they are orphaned and not mounted.
Files:
src/server/services/mcp/allowedTools.ts
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.{ts,tsx,js,jsx}: Place imports at the top of modules; use inline imports only for a documented circular dependency.
Run linting and ensure typecheck-related CI checks pass before submitting changes.
For UI or server changes, manually smoke-test development startup, recipe loading/building, validation banners, and live execution when execution behavior is touched.
Files:
src/server/services/mcp/allowedTools.ts
**/*.{js,jsx,ts,tsx,json,md,yaml,yml}
📄 CodeRabbit inference engine (CLAUDE.md)
Use
pnpmfor project package management and commands; do not usenpm install, which is blocked by a preinstall guard.
Files:
src/server/services/mcp/allowedTools.ts
**/*.{ts,tsx}
📄 CodeRabbit inference engine (CLAUDE.md)
**/*.{ts,tsx}: Account for React 19 and Vite 8 breaking changes rather than assuming conventions from earlier major versions; consult current documentation when API shapes are uncertain.
Do not trigger live Olive execution or batch runs in CI or virtual machines; limit CI validation to CPU-only recipe building, JSON export, and validation.Resolve ESLint errors; warnings are acceptable within the configured maximum of 20, but non-zero lint failures and reported errors must not be ignored.
Files:
src/server/services/mcp/allowedTools.ts
**/*.{js,jsx,ts,tsx,json}
📄 CodeRabbit inference engine (AGENTS.md)
Use pnpm 11.17 as the package manager; do not use npm install because the preinstall guard blocks it.
Files:
src/server/services/mcp/allowedTools.ts
src/server/**/*.ts
📄 CodeRabbit inference engine (AGENTS.md)
Use the server testing configuration and targeted server tests for changes under
src/server/; integration tests must use the existing mocks for child processes, AI providers, and fetch.
Files:
src/server/services/mcp/allowedTools.ts
olive-mcp-server/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
olive-mcp-server/**/*.py: Use Python 3.10 or newer for the Olive MCP server and keep themcpdependency pinned below version 2.
Do not trigger real Olive optimization or batch runs in CI, tests, or VM environments; use CPU-only recipe building, JSON export, and validation flows instead.
Files:
olive-mcp-server/tests/test_agent_lazy_import.pyolive-mcp-server/tests/test_agent_planner_props.pyolive-mcp-server/olive_mcp_server/tools/__init__.pyolive-mcp-server/tests/test_agent_diagnosis_props.pyolive-mcp-server/olive_mcp_server/mcp_server.pyolive-mcp-server/tests/test_call_tool.pyolive-mcp-server/olive_mcp_server/tools/agent_model_info.pyolive-mcp-server/tests/test_agent_compare.pyolive-mcp-server/tests/test_agent_model_info_props.pyolive-mcp-server/olive_mcp_server/tools/agent_planner.pyolive-mcp-server/tests/test_agent_error_structure.pyolive-mcp-server/olive_mcp_server/tools/agent_execute.pyolive-mcp-server/olive_mcp_server/tools/agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute_props.pyolive-mcp-server/olive_mcp_server/tools/agent_compare.pyolive-mcp-server/tests/test_agent_compare_props.pyolive-mcp-server/tests/test_agent_planner.pyolive-mcp-server/tests/test_agent_model_info.pyolive-mcp-server/tests/test_agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute.py
olive-mcp-server/tests/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Run MCP server tests with pytest using
python -m pytest tests -qfromolive-mcp-server.
Files:
olive-mcp-server/tests/test_agent_lazy_import.pyolive-mcp-server/tests/test_agent_planner_props.pyolive-mcp-server/tests/test_agent_diagnosis_props.pyolive-mcp-server/tests/test_call_tool.pyolive-mcp-server/tests/test_agent_compare.pyolive-mcp-server/tests/test_agent_model_info_props.pyolive-mcp-server/tests/test_agent_error_structure.pyolive-mcp-server/tests/test_agent_execute_props.pyolive-mcp-server/tests/test_agent_compare_props.pyolive-mcp-server/tests/test_agent_planner.pyolive-mcp-server/tests/test_agent_model_info.pyolive-mcp-server/tests/test_agent_diagnosis.pyolive-mcp-server/tests/test_agent_execute.py
🧠 Learnings (1)
📚 Learning: 2026-08-10T03:41:03.611Z
Learnt from: tonythethompson
Repo: tonythethompson/Olive-Studio PR: 203
File: src/components/features/input/GitHubRecipeSync.tsx:5-5
Timestamp: 2026-08-10T03:41:03.611Z
Learning: In the Olive-Studio repository, treat imports from the `@/components/ui` barrel as conforming to the established UI import convention. Do not flag these imports solely because a general guideline prefers importing from concrete modules.
Applied to files:
src/server/services/mcp/allowedTools.ts
🪛 ast-grep (0.45.0)
olive-mcp-server/tests/test_agent_lazy_import.py
[error] 30-35: Command coming from incoming request
Context: subprocess.run(
[sys.executable, "-c", script],
check=False,
capture_output=True,
text=True,
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').
(subprocess-from-request)
olive-mcp-server/tests/test_agent_planner_props.py
[warning] 68-68: Regex pattern passed to re is built from a non-literal (variable, call, concatenation, or f-string) value. If that value is attacker-controlled it can introduce a malicious pattern with catastrophic backtracking (ReDoS). Use a hardcoded literal pattern, or validate/escape untrusted input with re.escape() and bound the regex complexity before compiling.
Context: re.search(rf"\b{re.escape(kw)}\b", lower)
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-python)
[warning] 81-81: Regex pattern passed to re is built from a non-literal (variable, call, concatenation, or f-string) value. If that value is attacker-controlled it can introduce a malicious pattern with catastrophic backtracking (ReDoS). Use a hardcoded literal pattern, or validate/escape untrusted input with re.escape() and bound the regex complexity before compiling.
Context: re.search(rf"\b{re.escape(kw)}\b", lower)
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-python)
[warning] 90-90: Regex pattern passed to re is built from a non-literal (variable, call, concatenation, or f-string) value. If that value is attacker-controlled it can introduce a malicious pattern with catastrophic backtracking (ReDoS). Use a hardcoded literal pattern, or validate/escape untrusted input with re.escape() and bound the regex complexity before compiling.
Context: re.search(rf"\b{re.escape(kw)}\b", lower)
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-python)
olive-mcp-server/tests/test_agent_diagnosis_props.py
[info] 113-113: use jsonify instead of json.dumps for JSON output
Context: json.dumps(recipe)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 122-122: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/olive_mcp_server/tools/agent_model_info.py
[warning] 160-160: Request-controlled URL passed to urlopen; validate against an allowlist to prevent SSRF.
Context: urllib.request.urlopen(req, timeout=_HF_TIMEOUT_SECONDS)
Note: [CWE-918] Server-Side Request Forgery (SSRF).
(urlopen-unsanitized-data)
[warning] 71-71: XPath query is request-/variable-derived; use parameterized XPath to prevent injection.
Context: _SIZE_TOKEN_RE.findall(model_id)
Note: [CWE-643] Improper Neutralization of Data within XPath Expressions ('XPath Injection').
(xpath-injection-python)
olive-mcp-server/tests/test_agent_compare.py
[info] 343-343: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 348-348: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 366-366: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 377-377: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_model_info_props.py
[info] 68-68: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 146-146: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/olive_mcp_server/tools/agent_planner.py
[warning] 98-98: Regex pattern passed to re is built from a non-literal (variable, call, concatenation, or f-string) value. If that value is attacker-controlled it can introduce a malicious pattern with catastrophic backtracking (ReDoS). Use a hardcoded literal pattern, or validate/escape untrusted input with re.escape() and bound the regex complexity before compiling.
Context: re.search(pattern, lower)
Note: [CWE-1333] Inefficient Regular Expression Complexity.
(redos-non-literal-regex-python)
olive-mcp-server/tests/test_agent_error_structure.py
[info] 58-58: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/olive_mcp_server/tools/agent_diagnosis.py
[info] 69-69: use jsonify instead of json.dumps for JSON output
Context: json.dumps(hardware_probe, separators=(",", ":"))
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 89-89: use jsonify instead of json.dumps for JSON output
Context: json.dumps(value)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_execute_props.py
[info] 147-147: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 169-169: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_compare_props.py
[info] 246-246: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_planner.py
[info] 84-84: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 125-125: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 173-173: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 217-217: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 255-255: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 300-300: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 351-351: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_model_info.py
[info] 31-31: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_diagnosis.py
[info] 48-48: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
[info] 361-361: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
olive-mcp-server/tests/test_agent_execute.py
[info] 27-27: use jsonify instead of json.dumps for JSON output
Context: json.dumps(result)
Note: [CWE-116] Improper Encoding or Escaping of Output.
(use-jsonify)
🪛 GitHub Check: CodeFactor
olive-mcp-server/olive_mcp_server/tools/agent_model_info.py
[warning] 148-148: olive-mcp-server/olive_mcp_server/tools/agent_model_info.py#L148
Audit url open for permitted schemes. Allowing use of file:/ or custom schemes is often unexpected. (B310)
olive-mcp-server/olive_mcp_server/tools/agent_planner.py
[notice] 344-469: olive-mcp-server/olive_mcp_server/tools/agent_planner.py#L344-L469
Complex Method
[notice] 164-248: olive-mcp-server/olive_mcp_server/tools/agent_planner.py#L164-L248
Complex Method
olive-mcp-server/olive_mcp_server/tools/agent_execute.py
[notice] 65-220: olive-mcp-server/olive_mcp_server/tools/agent_execute.py#L65-L220
Complex Method
olive-mcp-server/olive_mcp_server/tools/agent_compare.py
[notice] 63-149: olive-mcp-server/olive_mcp_server/tools/agent_compare.py#L63-L149
Complex Method
🪛 Ruff (0.16.1)
olive-mcp-server/tests/test_agent_lazy_import.py
[warning] 21-29: Use f-string instead of format call
Convert to f-string
(UP032)
olive-mcp-server/olive_mcp_server/tools/agent_compare.py
[warning] 53-53: Name compared with itself, consider replacing f != f
(PLR0124)
[warning] 104-107: Combine if branches using logical or operator
Combine if branches
(SIM114)
[warning] 106-109: Combine if branches using logical or operator
Combine if branches
(SIM114)
olive-mcp-server/tests/test_agent_execute.py
[warning] 17-17: Use from olive_mcp_server.tools import agent_execute in lieu of alias
Replace with from olive_mcp_server.tools import agent_execute
(PLR0402)
🔍 Remote MCP GitHub Copilot
Relevant review context
- PR
#245is open with 89 commits, 22 files changed, and +4,899/-3. Its mergeable state is currentlyunstable. - Checks: CodeQL, validation, security, Docker build, and Olive pass availability succeeded; Python tests were still in progress, while CodeFactor failed at retrieval time.
- The PR consolidates earlier work from PRs
#225–#241, including the planner, diagnosis, comparison, model-info, lazy-import, and Hypothesis-testing changes.
Important implementation contracts
- The current head registers all five tools in the Python MCP registry and allowlist.
GET /api/olive/agent/status/:jobIdexposeslatestMetrics, but repository code defines this field as GPU sampling telemetry (GpuMetrics). The comparison tool now excludes jobs with no matching metrics, but the source of optimization latency/size/accuracy results should still be verified./api/olive/jobs/validateexists and returns avalidfield, matchingdiagnose_and_fix’s validation check.- Remote-code handling is currently opt-in: defaults use
trustRemoteCode: false, recipe generation emitstrust_remote_codeonly when explicitly true, and the troubleshooting KB marks the remote-code fix as non-applyable. - Earlier automated comments about stray TypeScript tokens, duplicate defaults, and duplicate recipe-builder endings are outdated; the current head files no longer contain those reported fragments.
🔇 Additional comments (19)
olive-mcp-server/olive_mcp_server/mcp_server.py (1)
14-20: LGTM!Also applies to: 96-116, 140-153
olive-mcp-server/olive_mcp_server/tools/__init__.py (1)
39-39: LGTM!olive-mcp-server/tests/test_agent_lazy_import.py (1)
1-37: LGTM!olive-mcp-server/tests/test_call_tool.py (1)
1-1: LGTM!Also applies to: 32-43
src/server/services/mcp/allowedTools.ts (1)
33-38: 🔒 Security & PrivacyConfirm /api/olive/jobs/submit honors agent access.
execute_and_observeis allowlisted and sends{"recipe": recipe}to/api/olive/jobs/submit, but the Studio route implementation is not visible in the availablesrc/server/routes/olive.tssubmission checks, which show/olive/jobs/validateand/olive/jobs/status. Confirm that/api/olive/jobs/submitrequiresallowJobSubmission === trueby the same Studio policy path assubmit_optimization_job.olive-mcp-server/olive_mcp_server/tools/agent_execute.py (1)
40-111: LGTM!olive-mcp-server/tests/test_agent_execute.py (1)
36-355: LGTM!olive-mcp-server/tests/test_agent_execute_props.py (1)
32-197: LGTM!olive-mcp-server/pyproject.toml (1)
21-21: LGTM!olive-mcp-server/tests/test_agent_diagnosis_props.py (1)
32-175: LGTM!Also applies to: 191-346
olive-mcp-server/tests/test_agent_planner.py (4)
22-33: The success mock returns{}, which mirrors the laxerror-key-only check. I raised the root cause and the mock update inolive-mcp-server/olive_mcp_server/tools/agent_planner.pyat Lines 319-336.
182-218: LGTM!
229-243: LGTM!Also applies to: 267-301
312-337: LGTM!Also applies to: 363-407
olive-mcp-server/olive_mcp_server/tools/agent_planner.py (4)
84-115: LGTM!
156-161: LGTM!
428-469: LGTM!
164-173: 📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
guideandmodel_typeare dead parameters, and the caller pays forguideanyway.Neither parameter is read in the body. The docstring claims the patch comes "from strategy and guide results", which is not true.
plan_optimizationstill callsget_hardware_optimization_guideat Line 413 and loads hardware profiles for a value that is discarded.Either consume the guide, for example by mapping
calibration_sizeandoptimal_batch_sizeintopasses, or drop both parameters and the call. The guide carries real planning data, so consuming it is the better outcome.♻️ Minimal cleanup if the guide is not needed yet
def _compose_ui_state_patch( strategy: dict[str, Any], - guide: dict[str, Any] | None, model_id: str, - model_type: str, hardware_target: str, hardware_probe: dict[str, Any] | None, intent: str, ) -> dict[str, Any]: - """Compose a UIState patch from strategy and guide results.""" + """Compose a UIState patch from the strategy result and hardware probe."""Then update the call site at Lines 418-426 and remove the unused
guidelookup at Lines 412-415.> Likely an incorrect or invalid review comment.olive-mcp-server/tests/test_agent_planner_props.py (1)
119-147: LGTM!
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
|
Addressed the linked CodeFactor complex-method findings in 086ede6 by extracting focused helpers for planner composition/response building, metric normalization/scoring, submission error mapping, polling state, and artifact extraction. The complete focused agent suite passes: 143 tests. |
- Default bind to 127.0.0.1 with OLIVE_BIND=0.0.0.0 opt-in and documented threat model - Apply loopback-only gating to /api/olive/run, status, stream, cancel - Enforce SYNC_KB_TOKEN on /api/mcp/sync-kb and return 500 on sync errors - Add global Express error handler to prevent stack-trace leaks - Tighten Tauri CSP to match Express CSP - Update AGENTS.md/.github/README.md/REVIEW.md and remove local-first branding - Drive-by: fix QnnAbiExecutionProvider exhaustiveness, UIState test object shape, and passMigration structuredClone to keep tsc/test green Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 44 out of 44 changed files in this pull request and generated no new comments.
Suppressed comments (4)
olive-mcp-server/olive_mcp_server/tools/agent_compare.py:26
_TERMINAL_STATESis defined but unused in this module. Removing it avoids dead code and reduces confusion about how terminal vs scoreable states are handled.
_VALID_PREFERENCES = frozenset({"latency", "size", "accuracy", "balanced"})
_JOB_ID_PATTERN = re.compile(r"^[A-Za-z0-9_-]{1,128}$")
_TERMINAL_STATES = frozenset({"completed", "failed", "cancelled"})
server.ts:301
- The startup log builds a URL using
displayHostthat can include spaces/parentheses (e.g.0.0.0.0 (all interfaces)), producing an invalid URL string. This makes the log misleading whenOLIVE_BINDis set to all interfaces.
const displayHost = IS_ALL_INTERFACES ? "0.0.0.0 (all interfaces)" : BIND_HOST;
// eslint-disable-next-line no-console -- intentional server startup message
console.log(`Server running on http://${displayHost}:${PORT}`);
if (IS_ALL_INTERFACES) {
console.warn(
server.ts:180
- The global Express error handler is registered before the Vite/static middleware is attached. In Express, an error handler only catches errors from middleware registered before it, so this will not sanitize errors thrown by the Vite middleware/static serving branch (despite the comment saying it's global).
// ─── Global error handling ────────────────────────────────────────────────────
// Sanitize 500s so stack traces are not leaked to clients.
app.use((err: unknown, _req: Request, res: Response, _next: NextFunction) => {
console.error("[express] unhandled error:", err instanceof Error ? err.stack ?? err.message : err);
if (res.headersSent) return;
olive-mcp-server/olive_mcp_server/tools/agent_planner.py:76
_HARDWARE_TO_PROVIDERis defined but never used. This adds dead code and can drift out of sync with_infer_providerover time.
# Provider mapping for UIState ihvProvider field
_HARDWARE_TO_PROVIDER: dict[str, str] = {
"nvidia": "nvidia",
"intel": "intel",
"qualcomm": "qualcomm",
…emote_code safety - Strip trust_remote_code auto-elevation in diagnose_and_fix and describe changes via diff instead of patch-only - Handle degenerate metric ranges (max == min) with neutral 0.5 score in compare_results - Penalize missing metrics with 0.0 score while preserving full weight to prevent score inflation - Validate job_id format and reject duplicates in compare_results - Catch ImportError in _resolve_tool for optional
Dismissed because a newer commit was pushed; Greptile will re-review the current head.
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Summary
Adds 5 new Python MCP tools for the autonomous agent optimization loop:
execute_and_observe— submit recipe + poll to terminal stateplan_optimization— NL intent → UIState patchdiagnose_and_fix— error diagnosis + RFC 7386 merge-patch recipe repaircompare_results— multi-job scoring with preference weightingget_model_info— HF metadata lookup with heuristic fallbackTesting
Spec
.kiro/specs/v0.3-agent-mcp-tools/(requirements → design → tasks)Consolidates PRs #225–#241 (close those after merge).