Skip to content

fix: duration not actually corerespond with video duration problem #54

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jaesm14774 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: duration not actually corerespond with video duration problem #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
171 changes: 109 additions & 62 deletions nodes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
import os
import torch
import json
from torchvision.transforms import v2
from accelerate import init_empty_weights
from accelerate.utils import set_module_tensor_to_device

Expand All @@ -28,69 +27,116 @@
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
log = logging.getLogger(__name__)

def process_video_tensor(video_tensor: torch.Tensor, duration_sec: float) -> tuple[torch.Tensor, torch.Tensor, float]:
def process_video_tensor(video_tensor: torch.Tensor, duration_sec: float, input_fps: float = None) -> tuple[torch.Tensor, torch.Tensor, float]:
"""
處理影片張量，轉換為 CLIP 和 SYNC 所需格式
當影片幀數不足時，會透過時間插值來擴充幀數以匹配指定的 duration

Args:
video_tensor: 影片張量 (frames, height, width, channels)
duration_sec: 期望的音訊時長（秒）
input_fps: 輸入影片的幀率（如果為 None 則自動計算）

Returns:
clip_frames: CLIP 模型用的幀
sync_frames: Synchformer 用的幀
actual_duration: 實際使用的時長（秒）
"""
_CLIP_SIZE = 384
_CLIP_FPS = 8.0

_SYNC_SIZE = 224
_SYNC_FPS = 25.0

# Synchformer 的分段參數（與 features_utils.py 保持一致）
_SYNC_SEGMENT_SIZE = 16
_SYNC_STEP_SIZE = 8
_SYNC_DOWNSAMPLE = 2

clip_transform = v2.Compose([
v2.Resize((_CLIP_SIZE, _CLIP_SIZE), interpolation=v2.InterpolationMode.BICUBIC),
v2.ToPILImage(),
v2.ToTensor(),
v2.ConvertImageDtype(torch.float32),
])

sync_transform = v2.Compose([
v2.Resize(_SYNC_SIZE, interpolation=v2.InterpolationMode.BICUBIC),
v2.CenterCrop(_SYNC_SIZE),
v2.ToPILImage(),
v2.ToTensor(),
v2.ConvertImageDtype(torch.float32),
v2.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])

# Assuming video_tensor is in the shape (frames, height, width, channels)
total_frames = video_tensor.shape[0]
clip_frames_count = int(_CLIP_FPS * duration_sec)
sync_frames_count = int(_SYNC_FPS * duration_sec)

# Adjust duration if there are not enough frames
if total_frames < clip_frames_count:
log.warning(f'Clip video is too short: {total_frames / _CLIP_FPS:.2f} < {duration_sec:.2f}')
clip_frames_count = total_frames
duration_sec = total_frames / _CLIP_FPS

if total_frames < sync_frames_count:
log.warning(f'Sync video is too short: {total_frames / _SYNC_FPS:.2f} < {duration_sec:.2f}, truncating to {total_frames / _SYNC_FPS:.2f} sec')
sync_frames_count = total_frames
duration_sec = total_frames / _SYNC_FPS

clip_frames = video_tensor[:clip_frames_count]
sync_frames = video_tensor[:sync_frames_count]

clip_frames = clip_frames.permute(0, 3, 1, 2)
sync_frames = sync_frames.permute(0, 3, 1, 2)

clip_frames = torch.stack([clip_transform(frame) for frame in clip_frames])
sync_frames = torch.stack([sync_transform(frame) for frame in sync_frames])

clip_length_sec = clip_frames.shape[0] / _CLIP_FPS
sync_length_sec = sync_frames.shape[0] / _SYNC_FPS

# if clip_length_sec < duration_sec:
# log.warning(f'Clip video is too short: {clip_length_sec:.2f} < {duration_sec:.2f}')
# log.warning(f'Truncating to {clip_length_sec:.2f} sec')
# duration_sec = clip_length_sec

# if sync_length_sec < duration_sec:
# log.warning(f'Sync video is too short: {sync_length_sec:.2f} < {duration_sec:.2f}')
# log.warning(f'Truncating to {sync_length_sec:.2f} sec')
# duration_sec = sync_length_sec

clip_frames = clip_frames[:int(_CLIP_FPS * duration_sec)]
sync_frames = sync_frames[:int(_SYNC_FPS * duration_sec)]

# 計算需要的幀數（使用指定的 duration）
clip_frames_needed = int(_CLIP_FPS * duration_sec)
sync_frames_needed = int(_SYNC_FPS * duration_sec)

if input_fps is not None and input_fps > 0:
video_actual_duration = total_frames / input_fps
log.info(f"輸入影片: {total_frames} 幀 @ {input_fps} fps (實際時長 {video_actual_duration:.2f}s)")
log.info(f"目標音訊時長: {duration_sec:.2f}s")
log.info(f"需要的幀數: CLIP={clip_frames_needed} @ {_CLIP_FPS} fps, SYNC={sync_frames_needed} @ {_SYNC_FPS} fps")

# 提取原始幀並轉換為 NCHW 格式
video_frames_nchw = video_tensor.permute(0, 3, 1, 2).contiguous() # (T, H, W, C) -> (T, C, H, W)

# === 處理 CLIP 幀 ===
if total_frames < clip_frames_needed:
# 需要時間插值來擴充幀數
log.info(f"CLIP: 影片幀數 {total_frames} < 需求 {clip_frames_needed}，使用時間插值擴充")
# 使用 3D 插值在時間維度擴充
# 添加 batch 維度: (T, C, H, W) -> (1, C, T, H, W)
video_5d = video_frames_nchw.permute(1, 0, 2, 3).unsqueeze(0) # (1, C, T, H, W)
# 時間插值
video_5d_interp = torch.nn.functional.interpolate(
video_5d,
size=(clip_frames_needed, video_5d.shape[3], video_5d.shape[4]),
mode='trilinear',
align_corners=False
)
# 轉回 (T, C, H, W)
clip_frames = video_5d_interp.squeeze(0).permute(1, 0, 2, 3) # (T, C, H, W)
else:
# 幀數足夠，直接截取
clip_frames = video_frames_nchw[:clip_frames_needed]

# 空間 resize 到 384x384
clip_frames = torch.nn.functional.interpolate(
clip_frames,
size=(_CLIP_SIZE, _CLIP_SIZE),
mode='bicubic',
align_corners=False
)

# === 處理 SYNC 幀 ===
if total_frames < sync_frames_needed:
# 需要時間插值來擴充幀數
log.info(f"SYNC: 影片幀數 {total_frames} < 需求 {sync_frames_needed}，使用時間插值擴充")
# 使用 3D 插值在時間維度擴充
video_5d = video_frames_nchw.permute(1, 0, 2, 3).unsqueeze(0) # (1, C, T, H, W)
video_5d_interp = torch.nn.functional.interpolate(
video_5d,
size=(sync_frames_needed, video_5d.shape[3], video_5d.shape[4]),
mode='trilinear',
align_corners=False
)
sync_frames = video_5d_interp.squeeze(0).permute(1, 0, 2, 3) # (T, C, H, W)
else:
# 幀數足夠，直接截取
sync_frames = video_frames_nchw[:sync_frames_needed]

# 空間處理：先 resize 到短邊 224，然後 center crop
h, w = sync_frames.shape[2], sync_frames.shape[3]
if h < w:
new_h, new_w = _SYNC_SIZE, int(_SYNC_SIZE * w / h)
else:
new_h, new_w = int(_SYNC_SIZE * h / w), _SYNC_SIZE

sync_frames = torch.nn.functional.interpolate(
sync_frames,
size=(new_h, new_w),
mode='bicubic',
align_corners=False
)

# Center crop
top = (new_h - _SYNC_SIZE) // 2
left = (new_w - _SYNC_SIZE) // 2
sync_frames = sync_frames[:, :, top:top+_SYNC_SIZE, left:left+_SYNC_SIZE]

# Normalize sync frames
sync_frames = sync_frames * 2.0 - 1.0 # [0, 1] -> [-1, 1]

log.info(f"✅ 處理完成: clip={clip_frames.shape[0]} 幀 ({clip_frames.shape[0]/_CLIP_FPS:.2f}s @ {_CLIP_FPS} fps), "
f"sync={sync_frames.shape[0]} 幀 ({sync_frames.shape[0]/_SYNC_FPS:.2f}s @ {_SYNC_FPS} fps), "
f"音訊時長={duration_sec:.2f}s")

return clip_frames, sync_frames, duration_sec

Expand Down Expand Up @@ -298,7 +344,7 @@ def INPUT_TYPES(s):
"required": {
"mmaudio_model": ("MMAUDIO_MODEL",),
"feature_utils": ("MMAUDIO_FEATUREUTILS",),
"duration": ("FLOAT", {"default": 8, "step": 0.01, "tooltip": "Duration of the audio in seconds"}),
"duration": ("FLOAT", {"default": 8, "step": 0.01, "tooltip": "期望的音訊時長（秒），實際時長會根據影片幀數自動調整"}),
"steps": ("INT", {"default": 25, "step": 1, "tooltip": "Number of steps to interpolate"}),
"cfg": ("FLOAT", {"default": 4.5, "step": 0.1, "tooltip": "Strength of the conditioning"}),
"seed": ("INT", {"default": 0, "min": 0, "max": 0xffffffffffffffff}),
Expand All @@ -309,6 +355,7 @@ def INPUT_TYPES(s):
},
"optional": {
"images": ("IMAGE",),
"input_fps": ("FLOAT", {"default": 16.0, "min": 1.0, "max": 60.0, "step": 0.01, "tooltip": "輸入影片的幀率（fps），用於計算實際時長"}),
},
}

Expand All @@ -317,7 +364,7 @@ def INPUT_TYPES(s):
FUNCTION = "sample"
CATEGORY = "MMAudio"

def sample(self, mmaudio_model, seed, feature_utils, duration, steps, cfg, prompt, negative_prompt, mask_away_clip, force_offload, images=None):
def sample(self, mmaudio_model, seed, feature_utils, duration, steps, cfg, prompt, negative_prompt, mask_away_clip, force_offload, images=None, input_fps=16.0):
device = mm.get_torch_device()
offload_device = mm.unet_offload_device()
rng = torch.Generator(device=device)
Expand All @@ -327,8 +374,8 @@ def sample(self, mmaudio_model, seed, feature_utils, duration, steps, cfg, promp

if images is not None:
images = images.to(device=device)
clip_frames, sync_frames, duration = process_video_tensor(images, duration)
print("clip_frames", clip_frames.shape, "sync_frames", sync_frames.shape, "duration", duration)
clip_frames, sync_frames, duration = process_video_tensor(images, duration, input_fps)
log.info(f"處理結果: clip_frames={clip_frames.shape}, sync_frames={sync_frames.shape}, duration={duration:.2f}s")
if mask_away_clip:
clip_frames = None
else:
Expand Down
95 changes: 95 additions & 0 deletions readme.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,3 +23,98 @@ Nvidia bigvganv2 (used with 44k mode)
https://huggingface.co/nvidia/bigvgan_v2_44khz_128band_512x

is autodownloaded to `ComfyUI/models/mmaudio/nvidia/bigvgan_v2_44khz_128band_512x`

# 重要更新（2025-11-17）

## ✨ 智能時間插值：讓短影片也能生成完整音訊

### 核心功能
**當影片幀數不足時，自動使用時間插值（temporal interpolation）擴充影片幀數，確保音訊時長與指定的 `duration` 完全匹配。**

這意味著即使您的影片只有 3 秒，也能生成 6 秒的音訊，影片會平滑地慢動作播放來填補時間。

### 問題背景
之前版本中，當影片幀數不足以支援指定的 `duration` 時，程式會自動縮短音訊時長，導致：
- 設定 `duration=6.31` 秒，實際只生成 3-4 秒音訊
- 影片後半段沒有聲音，體驗不佳

**原因分析**：
- 101 幀 @ 16 fps = 6.31 秒影片時長
- 但 MMAudio 的 Sync 模型需要 @ 25 fps，即需要 158 幀
- 幀數不足 → 被迫縮短時長 → 音訊只有 3.84 秒

### 解決方案：智能時間插值

現在採用 **trilinear interpolation（三線性插值）** 在時間維度上擴充影片幀數：

1. **自動檢測幀數不足**：當 `total_frames < required_frames` 時觸發
2. **平滑時間插值**：使用 PyTorch 的 3D 插值功能，在時間軸上生成中間幀
3. **保持視覺流暢**：插值後的影片看起來像慢動作，沒有突兀的跳幀

### 技術細節

```python
# 當 101 幀不足以支援 6.31 秒 @ 25 fps (需要 158 幀) 時
# 使用 3D 插值擴充：
video_5d = video_frames.unsqueeze(0) # (1, C, T, H, W)
interpolated = F.interpolate(
video_5d,
size=(158, H, W), # 擴充時間維度從 101 → 158
mode='trilinear', # 三線性插值
align_corners=False
)
# 結果：平滑生成 57 個中間幀
```

### 使用說明

**基本使用**（推薦）：
```
duration: 6.31 # 想要的音訊時長
input_fps: 16.0 # 影片實際幀率（ComfyUI 預設）
```

**結果**：
- ✅ 自動檢測影片只有 101 幀（6.31 秒 @ 16 fps）
- ✅ 自動插值擴充到 CLIP 需要的 50 幀 @ 8 fps
- ✅ 自動插值擴充到 SYNC 需要的 158 幀 @ 25 fps
- ✅ 生成完整的 6.31 秒音訊

**注意事項**：
- 插值會讓影片看起來像慢動作（但通常不明顯）
- 建議 `duration` 不要超過影片實際時長太多（2 倍以內效果最佳）
- 如果影片幀數充足，不會進行插值，直接使用原始幀

### 效能優化

1. **GPU 加速的時間插值**：使用 PyTorch 內建的 trilinear interpolation，充分利用 GPU
2. **批次處理**：所有幀一次性處理，避免逐幀操作
3. **記憶體高效**：使用 `contiguous()` 優化記憶體佈局

### 範例日誌輸出

**情境 1：幀數不足，需要插值**
```
輸入影片: 101 幀 @ 16.0 fps (實際時長 6.31s)
目標音訊時長: 6.31s
需要的幀數: CLIP=50 @ 8.0 fps, SYNC=158 @ 25.0 fps
SYNC: 影片幀數 101 < 需求 158，使用時間插值擴充
✅ 處理完成: clip=50 幀 (6.25s @ 8.0 fps), sync=158 幀 (6.32s @ 25.0 fps), 音訊時長=6.31s
```

**情境 2：幀數充足，不需插值**
```
輸入影片: 200 幀 @ 16.0 fps (實際時長 12.5s)
目標音訊時長: 6.0s
需要的幀數: CLIP=48 @ 8.0 fps, SYNC=150 @ 25.0 fps
✅ 處理完成: clip=48 幀 (6.0s @ 8.0 fps), sync=150 幀 (6.0s @ 25.0 fps), 音訊時長=6.0s
```

### 參數說明

- **`duration`**：期望的音訊時長（秒），現在會嚴格遵守此參數
- **`input_fps`**（可選，預設 16.0）：輸入影片的幀率，用於計算和顯示資訊
- 16 fps：ComfyUI 預設
- 24 fps：電影標準
- 30 fps：常見影片格式
- 60 fps：高幀率影片