fix(gemini): convert PDF to text in functionResponse parts to prevent API errors - #2035
Closed
echoVic wants to merge 4 commits into
Closed
fix(gemini): convert PDF to text in functionResponse parts to prevent API errors#2035echoVic wants to merge 4 commits into
echoVic wants to merge 4 commits into
Conversation
added 4 commits
March 1, 2026 11:19
QwenLM#2025) - Trim skill name in validateToolParams to handle cases where model adds extra whitespace - Update params.skill with trimmed value for consistent usage throughout the tool - Add test cases for leading/trailing whitespace and Chinese skill names Fixes QwenLM#2025
… API errors (QwenLM#2020) - Add PDF (application/pdf) to unsupported media types in convertUnsupportedMediaToText - This prevents 'Invalid value: file' API errors when reading PDF files - PDF content in tool responses is now converted to explanatory text - Add test cases for PDF inlineData and fileData conversion Fixes QwenLM#2020
…l names (QwenLM#2012) - Add 'bash' and 'sh' to ToolNamesMigration as aliases for run_shell_command - Migrate legacy tool names in CoreToolScheduler before registry lookup - This fixes 'Tool "bash" not found in registry' errors when models use common aliases - Add test case for tool name migration Fixes QwenLM#2012
echoVic
requested review from
DennisYu07,
LaZzyMan,
Mingholy,
gwinthis,
pomelo-nwu and
tanzhenxin
as code owners
March 2, 2026 02:04
This was referenced Mar 2, 2026
Collaborator
LaZzyMan
added a commit
that referenced
this pull request
Jun 26, 2026
The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to 0.6.8 during an earlier sync-script trial whose code delta was not kept. Left as-is it would make a future sync diff 0.6.8->newer and silently skip the real 0.6.7->0.6.8 fixes. Correct it back to 0.6.7. Also record the four not-yet-merged upstream PRs we carry as cherry-picks (trycua/cua#2021/#2025/#2035/#2036) in .vendored-patches.md, and have sync-from-upstream.sh point at it so the next sync reconciles them.
shenyankm
pushed a commit
to shenyankm/qwen-code
that referenced
this pull request
Jun 26, 2026
…coordinates (QwenLM#5896) * feat(cua-driver): vendor trycua/cua driver with 1000-normalized coordinate support Vendor libs/cua-driver from trycua/cua into packages/cua-driver as the basis for qwen-code's computer-use backend, adding an opt-in relative (1000x1000 normalized) coordinate mode for Qwen-VL clients. - coord_norm.rs: 0-1000 <-> pixel conversion, per-(pid,window_id) size cache, tools/list description rewrite (TDD, 27 tests) - ToolRegistry: normalized field + invoke input/output hooks - protocol.rs: system-instruction coordinate wording switched by mode - serve.rs: daemon list path description rewrite (input_schema aware) - main.rs: CUA_DRIVER_RS_COORDINATE_SPACE env seed Default coordinate_space=pixels => zero behavior change for existing pixel clients. Set CUA_DRIVER_RS_COORDINATE_SPACE=normalized_1000 to enable. Excludes rust/target build output. * feat(cua-driver): make normalized coordinate scale configurable Add CUA_DRIVER_RS_COORDINATE_SCALE (default 1000) so the normalization full-scale can absorb the Qwen 999-vs-1000 cookbook ambiguity without a recompile. norm_to_px/px_to_norm now take an explicit scale; denormalize_args reads the process-wide COORDINATE_SCALE seeded once at startup from env. * ci(cua-driver): add cross-platform release workflow for vendored driver Standalone GitHub Action that builds, signs, and releases the vendored cua-driver under packages/cua-driver. Adapted from upstream trycua/cua cd-rust-cua-driver.yml: - macOS: universal binary (lipo arm64+x86_64), codesigned + notarized into CuaDriver.app using qwen-code's existing secrets (MAC_CSC_LINK cert + App Store Connect API key notarization); Developer ID identity is auto-discovered from the imported cert. - Linux: x86_64 + arm64, built in debian:11 for a glibc 2.31 floor. - Windows: x86_64 + arm64, unsigned (no EV cert, matches upstream). - Release: softprops/action-gh-release on cua-driver-rs-v* tags or manual dispatch, prerelease. Triggered by tag push (cua-driver-rs-v*) or workflow_dispatch. * chore(cua-driver): rebrand vendored driver as qwen-cua-driver Rename the vendored trycua/cua driver so the fork installs and runs independently of any upstream trycua install: - binary cua-driver -> qwen-cua-driver - bundle CuaDriver.app -> QwenCuaDriver.app - bundle id com.trycua.driver -> com.qwencode.cua-driver Updates the cargo/uia manifests, Info.plist, bundle/proxy launch paths, permission/health-report wording, the install/build scripts, and the cross-platform release workflow. * feat(cua-driver): finish relative-coordinate mode — toggle, scale, zoom/move_cursor - CUA_DRIVER_RS_COORDINATE_SPACE is now a 1/0 toggle (via is_env_truthy); default off keeps pixel mode byte-identical to upstream. - Thread CUA_DRIVER_RS_COORDINATE_SCALE through every coordinate surface (was hardcoded 1000): input denormalization already used it; now the rewritten screenshot dims, the tool/param descriptions, and the agent instructions track the configured scale too. - Normalize zoom (window basis) and move_cursor (screen basis) inputs and rewrite their descriptions, alongside click/double_click/right_click/drag. - Fix zoom on downscaled (Retina) windows: apply the get_window_state resize ratio so the crop lands on the region the agent saw. Normalized mode only; pixel-mode zoom unchanged. All coordinate behavior stays gated on the normalized flag, so the default (pixels) path is unchanged from upstream. * chore(cua-driver): add upstream-sync script (git subtree unusable here) `git subtree split --prefix=libs/cua-driver` hangs on a commit deep in trycua/cua's history, so the subtree add/pull workflow isn't usable for the vendored driver (and a pull would re-split + re-hang every time). Add scripts/sync-from-upstream.sh instead: it git-diffs two upstream refs (never walks the full history, so it dodges the hang), reprefixes the libs/cua-driver delta to packages/cua-driver, and `git apply --reject`s it on top of our local changes — conflicts land as *.rej for manual fixup. Record the vendored version in .vendored-from and document the migration + sync method in the design doc. * chore(cua-driver): exclude vendored driver from qwen-code ESLint The vendored packages/cua-driver tree carries upstream JS (e.g. the test-harness Electron app) that doesn't follow qwen-code's lint rules and fails CI. It is not a workspace package (no package.json) and is not qwen-code TypeScript, so add it to eslint.config.js global ignores — alongside packages/desktop/** — the standard treatment for vendored code. * fix(cua-driver): let start_session revive an idle-reaped session Ports the fix from upstream trycua/cua#2035 into the vendored driver. When a session is reaped for idleness, a subsequent start_session with the same id failed instead of resuming it. Revive the ended session in place so the agent can continue rather than getting a hard error. * fix(cua-driver): retry daemon socket writes on EAGAIN Ports the fix from upstream trycua/cua#2036 into the vendored driver. A non-blocking daemon socket can return EAGAIN/EWOULDBLOCK mid-write when the peer's receive buffer is momentarily full. The driver treated that as fatal and dropped the connection. Add a bounded retry/poll loop (mirror of the read-side socket_io helper) so transient back-pressure no longer kills the session; only a real timeout or hard error fails the write. * fix(cua-driver/linux): stop reporting bare "Clicked" for X11 synthetic clicks Ports the fix from upstream trycua/cua#2025 into the vendored driver. On X11, clicks are delivered via XSendEvent synthetic events, which many toolkits (GTK/SDL/Allegro) ignore because send_event is set. The driver still reported a flat success ("Clicked"), masking that nothing happened. Report the synthetic-delivery caveat honestly so the agent can fall back instead of assuming the click landed. (platform-linux crate is not built on macOS; verified by clean upstream apply and covered by upstream + release-workflow Linux CI.) * fix(cua-driver/windows): list empty-/null-title top-level windows Ports the fix from upstream trycua/cua#2021 into the vendored driver. list_windows filtered out any top-level window whose title was empty or null, so legitimate targets (splash screens, some Electron/game windows, tool windows) were invisible to the agent and unclickable. Include empty-title windows, using class name / process as a fallback label. (platform-windows crate is not built on macOS; verified by clean upstream apply and covered by upstream + release-workflow Windows CI.) * chore(cua-driver): track cherry-picked upstream PRs; fix vendored-from The vendored copy is actually at cua-driver-rs-v0.6.7 (workspace version and all 0.6.7->0.6.8 delta files confirm it), but .vendored-from had drifted to 0.6.8 during an earlier sync-script trial whose code delta was not kept. Left as-is it would make a future sync diff 0.6.8->newer and silently skip the real 0.6.7->0.6.8 fixes. Correct it back to 0.6.7. Also record the four not-yet-merged upstream PRs we carry as cherry-picks (trycua/cua#2021/QwenLM#2025/QwenLM#2035/QwenLM#2036) in .vendored-patches.md, and have sync-from-upstream.sh point at it so the next sync reconciles them. * ci(cua-driver): satisfy repo yamllint on the release workflow The vendored-driver release workflow tripped 114 quoted-strings violations under the repo's .yamllint (quote-type: single, required). Single-quote all string scalars to match every other workflow in .github/workflows. While reformatting, the release-notes body also got its paragraph blank lines collapsed and still referenced the old CUA_DRIVER_RS_COORDINATE_SPACE= normalized_1000 value — restore the blank lines and update it to the current 0/1 toggle (default 0 = off; optional CUA_DRIVER_RS_COORDINATE_SCALE=1000). * chore(cua-driver): sync vendored driver to cua-driver-rs-v0.6.8 First real run of scripts/sync-from-upstream.sh: it 3-way-applied the upstream 0.6.7->0.6.8 delta onto our local fork. 10/12 files applied cleanly; the 2 rejects (install.ps1, _install-rust.sh) were already-applied baked-version bumps (0.6.6->0.6.7, our copies were already at 0.6.7), i.e. no real conflict. 0.6.8 brings: Wayland input path (platform-linux), linux health_report + overlay tweaks, a platform-macos build.rs step, and dependency bumps. Version moved to 0.6.8 across the workspace. Verified our work survived the sync untouched: the relative-coordinate shim (coord_norm/protocol) and all four cherry-picked PRs (socket_io/session + linux/windows) are intact — in particular the 0.6.8 edit to platform-linux tools/impl_.rs landed alongside our QwenLM#2025 change with no collision. macOS cargo check + 132 core tests green. (platform-linux/windows + the binary integration test build only on their own runners; upstream CI covers those.) * ci(cua-driver): add a dry_run gate to the release workflow Mirror the desktop-release / release dry-run pattern: a workflow_dispatch dry_run boolean input (default true). The cross-platform build + package jobs always run and upload their artifacts; the GitHub Release job now publishes only on a tag push or an explicit dry_run=false dispatch. Lets us rehearse the whole build/package pipeline (dry_run=true, notarize=false) and inspect the produced artifacts without cutting a release. A branch push (no tag, not a dispatch) likewise builds without releasing.
5 tasks
AmariahAK
pushed a commit
to AmariahAK/qwen-code
that referenced
this pull request
Jul 3, 2026
* feat(cua-driver): sync vendored cua-driver from 0.6.8 to 0.7.0 Upstream 0.7.0 brings action-time modality (get_window_state always returns AX tree + screenshot, capture_mode retired), honest verification (every action returns verified/path/effect/escalation), browser page (CDP) tools, desktop-scope (capture_scope=window|desktop, get_desktop_state, screen-absolute click/scroll), and Rust-only consolidation (Swift backend retired). Cherry-pick patch changes: - QwenLM#2025 (X11 synthetic click): retired — superseded by 0.7.0 honest-verification - QwenLM#2035 (session revive): retired — superseded by 0.7.0 revive_session/loud-reject/is_session_lifecycle_tool - QwenLM#2036 (EAGAIN socket write retry): re-injected — merged upstream post-0.7.0 tag, EAGAIN os error 35 is not acceptable - QwenLM#2021 (null/empty-title windows + resolve_uwp_host_window): retained, adapted to 0.7.0's new is_listable_top_level and window_title helpers Shim extensions: - coord_norm: desktop-scope support — denormalize_args falls back to screen basis when no window cache (desktop-scope click), normalize_result handles get_desktop_state, ingest_screen_size reads get_desktop_state's screen_width/height - serve.rs: rewrite_coord_desc re-injected at both tools/list sites Install scripts simplified to Rust-only (Swift backend retired upstream). Fix platform-linux cross-compilation: cfg-gate wayland::PORTAL_LIBEI_ENABLED reference in health_report.rs for non-linux hosts. * fix(cua-driver): distinguish desktop-scope from window cache miss in denormalize_args When screenshot_w == 0 (no cached window size), check whether the call has a pid/window_id target before falling back to screen-size basis. Window-scoped clicks with a cache miss now correctly leave coordinates as-is instead of mapping them against the screen dimensions. * fix(cua-driver): use screenshot pixels not logical points for desktop-scope coord basis ingest_screen_size was reading screen_width/height (logical display points from CGDisplayBounds) for get_desktop_state, but the model operates on the screenshot PNG whose dimensions are physical pixels. On Retina displays (2x) this caused every desktop-scope click to land at half the intended position. Read screenshot_width/height instead — the same basis normalize_result rewrites to 1000. Updated test to use realistic Retina values (screen 1920, screenshot 3840) to catch 1:1 masking. * fix(cua-driver): split screen cache into logical (move_cursor) and physical (desktop-scope) The single screen_cache was being written by both get_screen_size (logical points, e.g. 1920x1080) and get_desktop_state (physical pixels, e.g. 3840x2160). Since move_cursor operates in CGEvent screen points (logical) while desktop-scope clicks operate in screenshot pixels (physical), whichever tool ran last would corrupt the other's denormalization basis — causing 2x overshoot or 1/2 offset on Retina. Split into: - SCREEN_SIZE: logical points, written by get_screen_size, consumed by move_cursor (screen_basis=true) - DESKTOP_SCREENSHOT_SIZE: physical pixels, written by get_desktop_state, consumed by desktop-scope click fallback (screenshot_w==0, no pid) Added cross-contamination isolation tests. --------- Co-authored-by: Shaojin Wen <shaojin.wensj@alibaba-inc.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2020
Converts PDF content to text description in functionResponse parts to prevent Gemini API 400 errors.
Changes
Testing