obs-vst: Fix incorrect VST window size on HiDPI display

[xlinshan]

Description

On Windows, the VST plugins' window sizes are rendered larger than the actual content on displays that have UI scale factor. The sizes are larger by the scale factor, for example, 100x100 content will have a 200x200 window on a 200% scaled screen, and 150x150 on a 150% scaled screen. This change adjust the window size to fit the content size. The screenshot is a loaded VST plugin with an extremely big window.

Motivation and Context

The problem described in the description above makes VST plugins difficult to use on HiDPI screen. This change makes the window as big as its content.
Issue #8092 talked about this problem. This change should fix it.

How Has This Been Tested?

I tested the change with VST2 plugins from MeldaProduction's MFreeFXBundle, and Youlean Loudness Meter 2. The MeldaProduction plugins are HiDPI aware plugins that adjust UI size according to OS setting, and the Youlean one is a HiDPI unaware plugin that provides a custom UI scale option. I loaded those plugins to see if they are created correctly, and I resized them, moved them from one display to another, and also changed the OS's UI scale factor while they are opened.

On a single display environment, the plugins are created and resized correctly under different UI scale factor settings. On a dual display environment, the plugins are created incorrectly if the main window of obs is on a sub-display with an UI scale different from the main display, that the window could be bigger or smaller than the content depending on the UI scale difference, but resizing, if applicable, works correctly and fixes the incorrect initial creation, and for HiDPI aware VST plugins, moving the window between the two displays fixes it too and the window adjusts to the different UI scale correctly.

The type of VST plugins that adjust its content to the window size (which is mentioned in #8092) is not tested due to my lack of access.

The test is done on a 2560x1600 laptop connecting to a 3840x2160 display on Windows 11 22h2.

Types of changes

Bug fix (non-breaking change which fixes an issue)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

Maintainer edit: Fixes #8092

[WizardCM]

To confirm, this change has only been tested on Windows?

[xlinshan]

To confirm, this change has only been tested on Windows?

Yes, the change has only been tested on Windows. I don't have a MacOS or Linux build environment set up, and I assume that the source file modified is platform specific, so I didn't have the thought. I will try to build it for MacOS (on Intel) and Linux (probably on Debian 11) to see if it has unintentional effects.

[xlinshan]

I tried to build it on macOS Monterey on an Intel MacBook and Ubuntu 22.04 on a VMware virtual machine.

On macOS, the modified obs-vst plugin works correctly with MeldaProduction plugins and Bias FX under every screen scale options.

On Ubuntu, obs-vst crashes obs when trying to open the window for a selected VST plugin. The error message on terminal is:

info: User selected new VST plugin: '/home/lbx/.lxvst/lsp-plugins/art-delay-mono.so'
qt.qpa.wayland: Wayland does not support QWindow::requestActivate()
X Error of failed request:  BadWindow (invalid Window parameter)
  Major opcode of failed request:  3 (X_GetWindowAttributes)
  Resource id in failed request:  0x7
  Serial number of failed request:  1407
  Current serial number in output stream:  1408
Segmentation fault (core dumped)

The obs installed from the ppa source has its VST plugin window opened without problems. However, the version that I built from the master branch of obsproject/obs-studio.git crashed too, with same error message. So I can't be sure if this commit has any negative effect on Linux platform.

[WizardCM]

On Windows, I can confirm current master does not handle scaling correctly, and this PR fixes the initial size on the primary display and subsequent resizes triggered by the VST. Tested with BlueCat, Voxengo SPAN, DSEQ3, and FabFilter.

I will do some further experimentation to try and improve behaviour on secondary screens (especially when dragging between displays) and I'll also test on other platforms.

It's worth noting that VST also has a pluginRespondsToDPIChanges flag that can be checked using effCanDo (which we don't currently use), which may be necessary for some VSTs (though I don't know what's special about it).

[PatTheMav]

@WizardCM given that it worked well for you in tests, would you recommend merging this? The change only affects Windows (Linux and macOS use a different source file) anyway.

[Tom-Evers]

There's something still not quite right with this. Please check my remark on 542cb87