Time stretching, pitch shifting known issues

[L3337]

We basically just repackage all of the open source time stretching / pitch shifting libraries, with the exception of:

• Time (Affecting Pitch)
• Pitch (Affecting Time)

, which simply re-pitch the audio in real time.

The open source libraries have the following known issues:

• Soundtouch: Usually pretty accurate on time, but sometimes the pitch is not what it was supposed to be, depending on the content. Also, sometimes it massively distorts the audio, usually changing to Rubberband is a better choice when this happens. Soundtouch is the default for reasons of rhythmic accuracy.
• Rubberband:
  • Generally sounds pretty good and is accurate on pitch, but often the accuracy of the length isn't 100%
  • Has issues with unicode characters in file paths on Windows. We are investigating fixing this, but it looks to be
    an exceedingly difficult and time consuming task
• Paulstretch: Not accurate on time, meant for turning sounds into atmospheres by stretching/smearing them by extreme amounts
• SBMS: Sometimes sounds strange, fairly accurate on pitch, often not very accurate on time, only works on Linux and Mac x86, does not work on ARM or Windows, where the option for it is hidden (fixed in our fork). We are maintaining our own fork of SBSMS that has many fixes and improvements to the original.

[yust1n]

I have to start stargate with LC_ALL=C otherwise I get

[2023-05-10 11:06:36,053] ERROR /opt/stargate/sglib/log.py : 105 - Traceback (most recent call last):

File "/opt/stargate/sgui/sgqt.py", line 411, in _editingFinished
value = self._str_to_value(value)

File "/opt/stargate/sgui/sgqt.py", line 258, in _str_to_value
value = float(_str)

ValueError: could not convert string to float: '1,2E+01'

[2023-05-10 11:06:40,946] ERROR /opt/stargate/sglib/log.py : 105 - Traceback (most recent call last):

File "/opt/stargate/sgui/sgqt.py", line 411, in _editingFinished
value = self._str_to_value(value)

File "/opt/stargate/sgui/sgqt.py", line 258, in _str_to_value
value = float(_str)

ValueError: could not convert string to float: '1,7E+02'

[2023-05-10 11:06:40,949] ERROR /opt/stargate/sglib/log.py : 105 - Traceback (most recent call last):

File "/opt/stargate/sgui/sgqt.py", line 272, in mousePressEvent
self.value_at_press = self.value()

File "/opt/stargate/sgui/sgqt.py", line 358, in value
return float(text)

ValueError: could not convert string to float: '1,2E+01'

[2023-05-10 11:06:53,830] ERROR /opt/stargate/sglib/log.py : 105 - Traceback (most recent call last):

File "/opt/stargate/sgui/wave_edit/init.py", line 72, in ok_handler
f_stretch = self.timestretch_amt.value()

File "/opt/stargate/sgui/sgqt.py", line 358, in value
return float(text)

ValueError: could not convert string to float: '1,2E+01'

Found another one ...
/opt/stargate/engine/sbsms ~/x.wav ~/y.wav 1.0 1.0 20.0 -20.0
gives me another wav compared to the one I get when I set the same parameters in stargate.
Btw I just copied the commandline from the terminalwindow where stargate is running.

The results also differ when I overwrite an existing file.

[L3337]

Hi @yust1n . What does running the locale command on the terminal show? I'm trying to replicate the issue locally, but I am not yet able to reproduce it. I created a branch called locale-fixes, please try building that locally and see if it fixes being able to run Stargate without LC_ALL=C.

Re: SBSMS issue: I'll look into that this weekend

[L3337]

Actually, just pushed another commit to the locale-fixes branch that should fix it. Apparently Qt was not taking the locale when it was set Python. Thanks for reporting, I have to wonder how many people encountered that and didn't report it.

[yust1n]

Seems all the issues are fixed in locale-fixes !!!!
So all you have to care for is replacement of existing files.
I am still in the middle of testing.
Replaced files are only 88bytes large. Looks like it's only the header without any data. Happens just sometimes.
I also managed to get a wav that still displays a waveform but is completly silent.
Hmmmm ... in an new project it displays as what it is - just empty.
Got a cache somewhere that doesn't get updated proper ???
I dared to rm some files in audio/samplegraph and now the project doesn't load anymore.
All I expected was a recreation of the now missing waveforms.

[L3337]

So all you have to care for is replacement of existing files.
I am still in the middle of testing.
Replaced files are only 88bytes large. Looks like it's only the header without any data. Happens just sometimes.
I also managed to get a wav that still displays a waveform but is completly silent.

Are you referring to time stretching? with SBSMS? Does it happen every time with the same file? If so, can you please provide an audio file that exhibits this behavior? Also, are you doing it in the wave editor or audio items? Thanks

[L3337]

@yust1n : Can you please test again with the latest main branch? Thanks! The locale-fixes broke Windows, so I had to make some changes. Then I found some problems testing on Korean Windows. This localization/internationalization stuff is really hard, LOL

[yust1n]

Stretching in the Wave Editor was the problem all the time. As far as I can tell everything works now in the main branch :) Where are the audio items for stretching hidden ???
Next thing to dive into is MIDI. It gets disconnected whenever i add FX for a synth in the Plugin Rack.
I have to turn it off and on in Hardware to make it work again. For sure we need to start another issue for this :D
I am using qjackctl and connect inputs to MidiThrough if you want to reproduce it.

[L3337]

As far as I can tell everything works now in the main branch

Glad it is working for your locale. This lead me to do some testing in Korean and Japanese Windows, where I discovered there are some pretty deep issues with non en-us locales that is going to require a lot of work. I'm tracking locale issues in #40

Where are the audio items for stretching hidden ???

$PROJECT_DIR/audio/timestretch

although in the wave editor, IIRC, it just exports then directly to any place you tell it to, the default location being:

$PROJECT_DIR/user/

Next thing to dive into is MIDI. It gets disconnected whenever i add FX for a synth in the Plugin Rack.

What do you mean "disconnected", it just stops working? Is qjackctl still showing it as connected? Is this another software providing MIDI events, or a hardware device being patched in using qjackctl?

[yust1n]

Yes, it's a hardwaredevice and it's connected via qjackctl midithrough. I need to turn it off and on in hardware to make it work again.

[L3337]

Thanks @yust1n , it's fixed in main branch, will be in the next release.

[L3337]

@yust1n : I believe that the SBSMS issues are fixed in the latest release. I ended up creating my own fork of SBSMS (which has been unmaintained for 10 years by the original author), and fixing many things in it. It was added as another git submodule, which may (or may not) require adjustment in the PKGBUILD for AUR.

[yust1n]

PKGBUILD is updated ;)

[luzpaz]

@L3337 is it possible for you to track what upstream issues each time stretching / pitch shifting library has that you're referring to? This may also help on-board developers to improve said situation.

Soundtouch https://gitlab.com/soundtouch/soundtouch
Rubberband https://github.com/breakfastquay/rubberband/issues
PaulXstretch https://github.com/essej/paulxstretch/issues
SBSMS https://github.com/stargatedaw/stargate-sbsms & https://github.com/claytonotey/libsbsms

[L3337]

Hi @luzpaz , good to hear from you.

That's a good idea, with some caveats, mostly around that the open source time-stretching/pitch-shifting libraries are almost completely unmaintained.

Perhaps it is time to consider:

• just doing what the other DAWs do, or
• doing as we do now and making multiple passes with the CLIs, adjusting the length passed as needed to try to get an accurate stretch, or

The first one is a fairly large undertaking. The 2nd is a bit of a hack, but only moderately difficult. The reason it is done this way now is so that we consume as little CPU as possible and be able to work on platforms such as Raspberry Pi 4/5. With some obvious tradeoffs that were accepted for the sake of keeping CPU usage very low.

https://gitlab.com/soundtouch/soundtouch

I have an upstream pull request with them to fix another issue that they are not accepting, which lead to me creating the Stargate DAW fork of it. The source repository seems to be in maintenance mode and has very little active development of the algorithm in recent years. The issues would probably need to be fixed in the Stargate fork, as it is unlikely to be accepted upstream.

https://github.com/breakfastquay/rubberband/issues

I definitely should at least submit an issue for the unicode CLI arguments bug. I have taken a TODO item to do that.

https://github.com/essej/paulxstretch/issues

We don't use that, that's something different. We use this which has not seen development in almost 10 years. It has been forked into the Stargate code base and is invoked by calling /usr/bin/stargate paulstretch ... (or the equivalent on macOS or Windows) with specific arguments. Once again, we would almost certainly need to fix any issues ourselves.

SBSMS https://github.com/stargatedaw/stargate-sbsms & https://github.com/claytonotey/libsbsms

SBSMS has been unmaintained for a very long time. The first link is our fork, the 2nd link is the fork for Audacity, this is the original. Once again, we would have to fix any issues in our fork.