[Core] Add Watermarking Framework & Support Watermarked Audio Generation w/ AudioSeal by alex-jw-brooks · Pull Request #7366 · vllm-project/vllm-omni

alex-jw-brooks · 2026-09-10T07:58:22Z

Purpose

Implements the core work for #7365

This adds a framework for watermarking multimodal outputs, and a first pass at integrating it in an endpoint / engine agnostic way. By default, the watermarking is off, and is configured at the engine level.

It also adds support for text watermarking through vLLM by forwarding the multimodal subconfig, as well as docs.

Test Plan

Added unit tests
Added integration tests for:
- stable audio, which uses the diffusion path and also produces lots of edge cases for AudioSeal (i.e., not 16khz, so ensures the generator/detector are okay in native SR without resampling, and produces stereo audio, which requires downmixing)
- Qwen3TTS, which hits the non diffusion path
  ^ Integration tests also load the AudioSeal detector and ensure the watermark was actually applied.

The AudioSeal generator is implemented with the streaming model, which is also used in the non-streaming case. I will post some benchmarking scripts for reproducibility, but it looked like the streaming model was pretty comparable in throughput to the non-streaming AudioSeal model, so it seems cleaner to just use it there.

Test Result

Tests pass. I also ran some validations checking that
- offline, realtime, chat completions, and audio gen look correct for watermarked vs not watermarked for the above & qwen3omni
- For those ^ that the watermarked results sound audibly identical to the not watermarked results
- And also tested both streaming and not streaming to ensure that the watermark is still correctly detected when streaming in chunks.

I still need to write some docs, so will make sure that there are examples for this in there

vllm-omni-review-bot · 2026-09-10T08:01:23Z

This PR appears to belong to: docs/design/module/observability.md, docs/design/module/input_output_modality_contracts.md.

Module owners: @lishunyang12 @vraiti @amy-why-3459

Routing: @lishunyang12 via module of the changed files; @vraiti via module of the changed files; @amy-why-3459 via semantic router

@alex-jw-brooks, please review your own changes and leave a short self-review comment describing what you checked. PRs without author self-review may not be assigned a reviewer.

Please take a look when you have a chance. If you would like an automated review, mention @vllm-omni-review-bot in a comment.

vllm-omni-review-bot · 2026-09-10T08:05:04Z

Omni ReviewBot triage note

Automated triage of commit dbb1b31ce26d produced:

Priority: high. Prompt maintainer attention is suggested.

These are automated triage suggestions only — the final decision belongs to the maintainers.

vllm-omni-review-bot · 2026-09-23T05:57:30Z

Omni ReviewBot: superseded

The CI failure noted on bb36db1c5072 refers to an earlier head; the pull request now points at c63b3ffb9acd.

vllm-omni-review-bot · 2026-09-28T19:28:51Z

Omni ReviewBot: superseded

The CI failure noted on c63b3ffb9acd refers to an earlier head; the pull request now points at 5490565213dd.

vllm-omni-review-bot · 2026-09-29T04:22:39Z

Omni ReviewBot: superseded

The CI failure noted on 5490565213dd refers to an earlier head; the pull request now points at 8dc48d4fd909.

vllm-omni-review-bot · 2026-09-29T07:33:24Z

Omni ReviewBot: superseded

The CI failure noted on 8dc48d4fd909 refers to an earlier head; the pull request now points at 1209ae916a4b.

hsliuustc0106 · 2026-10-04T03:23:38Z

@alex-jw-brooks this PR is labeled ready + high priority, but CI is failing on the latest commit:

docs/readthedocs.org:vllm-omni

Could you please take a look and push an update to get CI green? Once the checks pass we can proceed with review/merge. Thanks!

vllm-omni-review-bot · 2026-10-05T07:17:24Z

Omni ReviewBot: superseded

The CI failure noted on 190734467e5b refers to an earlier head; the pull request now points at fd2e9d92819e.

vllm-omni-review-bot · 2026-10-06T20:04:08Z

Omni ReviewBot routing record

Assigned Strict on zcode (GLM-5.3-Flash) under experiment fleet-strict-cursor-grok46-zcode-glm53flash-5050-c5-z10-20261002.

Signed-off-by: Alex Brooks <albrooks@redhat.com>

Signed-off-by: Alex Brooks <albrooks@redhat.com> fix headless flag Signed-off-by: Alex Brooks <albrooks@redhat.com>

Signed-off-by: Alex Brooks <albrooks@redhat.com>

linyueqian

Thanks for the follow-ups, approving at dbb1b31.

Opt-out: agreed. Matching vLLM's per-request opt-out API is reasonable, and the docs already say deployments that need watermarking should restrict opt-out at ingress.

Engine args: thanks, overriding watermark_config only when the Omni config yields a value keeps existing per-stage configs intact on the default-off path.

Throughput: thanks for measuring it. Since watermarking is off by default, batching the watermarker in a follow-up PR sounds good to me. It would help to link a tracking issue here so the TTFA numbers and the plan stay together.

alex-jw-brooks · 2026-10-28T02:08:27Z

Follow-up issue to track batching optimization here: #9501, thanks!

…gine Main (vllm-project#7366) now passes watermarking= to diffusion_engine.generate() from the pure-diffusion chat path. The test's fake engine had a fixed signature, so every request failed with an ErrorResponse once the PR was merged with main in CI. Accept and ignore extra keyword arguments. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: princepride <wangzhipeng628@gmail.com>

alex-jw-brooks requested review from Gaohan123, NickCao, Sy0307, amy-why-3459, hsliuustc0106, linyueqian, lishunyang12, tzhouam, yenuo26 and ywang96 as code owners September 10, 2026 07:58

hsliuustc0106 added core related to core module: cache, scheduler, engine, worker, modelrunner tts code related to tts models enhancement New feature or request high priority high priority issue, needs to be done asap labels Sep 11, 2026

hsliuustc0106 mentioned this pull request Sep 14, 2026

[RFC]: Add Multimodal Output Watermarking #7365

Open

1 task done

linyueqian added the ready label to trigger buildkite CI label Sep 14, 2026

lishunyang12 reviewed Sep 15, 2026

View reviewed changes

Comment thread vllm_omni/engine/stage_pool.py Resolved

Comment thread vllm_omni/watermarking/audio_seal.py Resolved

Comment thread pyproject.toml Resolved

alex-jw-brooks force-pushed the watermark branch from 95c9271 to bb36db1 Compare September 21, 2026 19:50

alex-jw-brooks force-pushed the watermark branch from bb36db1 to c63b3ff Compare September 28, 2026 19:25

alex-jw-brooks force-pushed the watermark branch from 8dc48d4 to 1209ae9 Compare October 5, 2026 05:15

alex-jw-brooks force-pushed the watermark branch from 1907344 to fd2e9d9 Compare October 9, 2026 04:24

alex-jw-brooks added 17 commits October 27, 2026 21:47

extract watermark utils & engine level per request opt out

70d92c7

Signed-off-by: Alex Brooks <albrooks@redhat.com>

strict mode watermarking core

6cc97f9

Signed-off-by: Alex Brooks <albrooks@redhat.com>

expose watermark opt out through server

5ce4dfd

Signed-off-by: Alex Brooks <albrooks@redhat.com>

fix test

d8cb0b5

Signed-off-by: Alex Brooks <albrooks@redhat.com>

rng fixes

8900b98

Signed-off-by: Alex Brooks <albrooks@redhat.com>

pin audioseal wm

17f4a7d

Signed-off-by: Alex Brooks <albrooks@redhat.com>

fix dropped arg

1fe1519

Signed-off-by: Alex Brooks <albrooks@redhat.com>

fix watermark handling on realtime

90807fc

Signed-off-by: Alex Brooks <albrooks@redhat.com>

minor

7917394

Signed-off-by: Alex Brooks <albrooks@redhat.com>

add missing diffusion path

cdc4851

Signed-off-by: Alex Brooks <albrooks@redhat.com>

revise stale comment for process inputs

e8807bc

Signed-off-by: Alex Brooks <albrooks@redhat.com>

doc fixes

a78ca94

Signed-off-by: Alex Brooks <albrooks@redhat.com>

forward text watermarking to vllm

4c498b4

Signed-off-by: Alex Brooks <albrooks@redhat.com> fix headless flag Signed-off-by: Alex Brooks <albrooks@redhat.com>

add watermarking docs

97ce94b

Signed-off-by: Alex Brooks <albrooks@redhat.com>

precommit, test init

9302a4c

Signed-off-by: Alex Brooks <albrooks@redhat.com>

handle vllm turning watermark to trinary arg

3b1d520

Signed-off-by: Alex Brooks <albrooks@redhat.com>

minor

672d4a1

Signed-off-by: Alex Brooks <albrooks@redhat.com>

alex-jw-brooks force-pushed the watermark branch from f729d7b to 672d4a1 Compare October 27, 2026 21:56

alex-jw-brooks added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 27, 2026

only set wm in engine args if it's not none

dbb1b31

Signed-off-by: Alex Brooks <albrooks@redhat.com>

alex-jw-brooks added ready label to trigger buildkite CI and removed ready label to trigger buildkite CI labels Oct 27, 2026

linyueqian approved these changes Oct 28, 2026

View reviewed changes

linyueqian merged commit 61a6ac2 into vllm-project:main Oct 28, 2026
10 of 11 checks passed

alex-jw-brooks mentioned this pull request Oct 28, 2026

Optimize Audio Watermarking #9501

Open

princepride mentioned this pull request Oct 28, 2026

[Bugfix] Honor stream=true in pure-diffusion chat completions #9452

Merged

acostea-ionos mentioned this pull request Oct 28, 2026

[RFC]: Add Pixel Seal as an image watermarking algorithm #9528

Open

1 task done

Conversation

alex-jw-brooks commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

vllm-omni-review-bot commented Sep 10, 2026

Uh oh!

vllm-omni-review-bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot triage note

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vllm-omni-review-bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

vllm-omni-review-bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

vllm-omni-review-bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

vllm-omni-review-bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

hsliuustc0106 commented Oct 4, 2026

Uh oh!

vllm-omni-review-bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Omni ReviewBot: superseded

Uh oh!

vllm-omni-review-bot commented Oct 6, 2026

Uh oh!

linyueqian left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alex-jw-brooks commented Oct 28, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

alex-jw-brooks commented Sep 10, 2026 •

edited

Loading

vllm-omni-review-bot commented Sep 10, 2026 •

edited

Loading

vllm-omni-review-bot commented Sep 23, 2026 •

edited

Loading

vllm-omni-review-bot commented Sep 28, 2026 •

edited

Loading

vllm-omni-review-bot commented Sep 29, 2026 •

edited

Loading

vllm-omni-review-bot commented Sep 29, 2026 •

edited

Loading

vllm-omni-review-bot commented Oct 5, 2026 •

edited

Loading