feat: use Clapper enhancers

[GeopJr]

Clapper 0.8 is out, so let's do #1171 again. Should probably rewrite the whole preview card logic.

fix: #1178
fix: #938
fix: #749
fix: #662
fix: #924

[Rafostar]

Thanks for still working on this (its been a while) 👍

[GeopJr]

I'm considering making it an optional flatpak plugin for now for two reasons:

1. size; going off by just the CI here, it went from 0.8 to 5 MB. I assume it will be more on flathub / when packaged right. That's not 'a lot' but it's still 5x the current one
2. updating; it will be easier to update ytdl outside of tuba, especially since youtube seems to break it often

(fwiw, size is not a clapper problem but mostly ytdl, ffmpeg etc I assume)

what do you think?

[Rafostar]

what do you think?

Well, Tuba is a special case where video playback from sites like YT is not its major feature. It just can make-do without (it can just pop a browser). So having it as a plugin in Tuba makes sense.

Ideally, the easiest to maintain/include for everyone would be having it packaged in a Flatpak extension shared among the apps. This way, would give only one place to maintain it. Currently there are 4 apps (including Clapper itself) that I know of which would like to use it. But then, the questions arise: Why wasn't yt-dlp ever packaged into extension if many apps use it? If we are packaging enhancers, why not package whole Clapper into a Flatpak extension?

I dunno what are the requirements for such an extension to be accepted in Flathub. But if single place to package/maintain it existed, that would certainly make things easier. Currently, we are just adding them to manifests of each app separately.

fwiw, size is not a clapper problem but mostly ytdl, ffmpeg etc I assume

ffmpeg is already an extension, so it shouldn't increase size here. Outside of yt-dlp and enhancers itself, I think libpeas contributes to the size notably. For some reason its not in GNOME runtime (even through it appears to be a GNOME-tech).

[GeopJr]

Why wasn't yt-dlp ever packaged into extension if many apps use it? If we are packaging enhancers, why not package whole Clapper into a Flatpak extension?

We can ask on the flathub/pak matrix rooms for advice I guess. Packaging clapper (lib) as an extension with enhancers and whatnot sounds good to me. ytdlp specifically needs constant updating and can render apps unusable if left unmaintained (e.g. Pipeline would probably stop working).

[GeopJr]

Unless I'm missing something, dynamically loading libraries on runtime in vala is beyond dreadful... (I'd have to use https://valadoc.org/gmodule-2.0/GLib.Module.html). Might have to bundle clapper anyway. I'm testing now a build without enhancers to see if the size difference is trivial, then I could make a tuba plugin with just the enhancers

[Rafostar]

Having at least enhancers as an extension would be nice start. Size aside, the more concerning problem is maintenance effort (compatibility and updates to yt-dlp) that would have to be done in each app. And they could be opt-in this way, since they are already dynamically loaded (within Clapper with libpeas).

[Rafostar]

Um, I think you might have accidentally removed stuff from not the manifest used by CI.

[GeopJr]

🤦🤦🤦 thanks!

[GeopJr]

816KB => 1.01MB

Sounds good to me.

the more concerning problem is maintenance effort (compatibility and updates to yt-dlp) that would have to be done in each app

I did some thinking on this and... can it be actually done? Flathub being strict on version matching, might render the whole 'update outside of apps' effort meaningless.

I mean, if Tuba depends on version 1.0.0 of the enhancers extension and then ytdlp breaks and you fix it, won't you have to publish 1.0.1? And won't apps need to bump the version manually?

Unless flathub allows us to be loose on patches so apps can depend on version 1.0 and patches get applied automatically

[Rafostar]

I mean, if Tuba depends on version 1.0.0 of the enhancers extension and then ytdlp breaks and you fix it, won't you have to publish 1.0.1? And won't apps need to bump the version manually?

I have never done a Flatpak extension, but... I guess, no?

AFAIU (from the little reserach I did), application just names the branch name as version (usually "stable") and patch downloads of updated versions take place separately from app that uses it. Like "ffmpeg-full" extension used here.

[Rafostar]

size; going off by just the CI here, it went from 0.8 to 5 MB. I assume it will be more on flathub / when packaged right. That's not 'a lot' but it's still 5x the current one
updating; it will be easier to update ytdl outside of tuba, especially since youtube seems to break it often

I did some thinking on this and... can it be actually done?

It can be done (and I made it), but its up to Flathub maintainers whether they accept it, see: flathub/flathub#6106

I have also tried it locally besides Clapper itself with a modified Tuba manifest that mounts enhancers as an extension. So yeah, it works.

[Rafostar]

Approved as a bundle. See updated Clapper Flathub README for how to use this extension: flathub/com.github.rafostar.Clapper

Note that it will probably take a few hours from now before this will be accessible after merge.

[GeopJr]

Woah! I didn't expect it to actually get approved, nice, thanks!

[GeopJr]

Tested with and without, everything seems to work as expected! No idea how to make GNOME Builder install the extensions, but outside of it it worked fine.

I'll remove the CLAPPER_0_8 stuff since it's out now and can require it directly.

[Rafostar]

One thing that still bugs me, is why we need to build libpeas ourselves? 🤔

It seems to be in 47 runtime as far as I see: https://gitlab.gnome.org/GNOME/gnome-build-meta/-/blob/gnome-47/elements/core-deps/libpeas.bst?ref_type=heads

[Rafostar]

I mean, last time I checked build was failing without libpeas, but maybe I was testing this before 47? (I do not remember)

[GeopJr]

gnome-build-meta is not just flatpak runtimes, only a handful of things are in them. The actual reason it's not included is:

We don't want to support them for various reasons (e.g. vte is not needed by most apps, libpeas has questionable stability and unclear relevance for flatpak apps, freerdp cannot really be considered part of the GNOME platform). Apps should just bundle these.

https://gitlab.gnome.org/GNOME/gnome-build-meta/-/issues/425#note_1271294

🤷

[Rafostar]

Ah, good find. Then having it be build before Clapper for apps that want enhancers will be necessary. Nothing more we can do about it.

[GeopJr]

Merging! With that, Clapper is now the default player (if available). Thanks!

TODO: Make the audio visualizer's controls match Clapper's

[Rafostar]

TODO: Make Clapper be able to be used as audio player/visualizer 😉

[Rafostar]

Make the audio visualizer's controls match Clapper

Alternatively, remember that Clapper offers individual widgets that bottom seek bar consists of. You have the freedom to design and do your own controls instead of using pre-made "simplecontrols" widget if you want.

So you can also customize Clapper to match your app (probably entirely from UI file). Well, I guess compat with GtkVideo that Tuba media viewer is designed/made around for might make it slightly harder/inconvenient.

[GeopJr]

I should have asked for a review from you when I was implementing it but forgot 😩 The audio visualizer requires gstreamer so I had to write everything, including controls. I made them match the GTK ones as Gtk.Video was the default.

https://github.com/GeopJr/Tuba/tree/main/src/Widgets/AudioPlayer

The main thing I needed was levels (https://github.com/GeopJr/Tuba/blob/main/src/Widgets/AudioPlayer/Stream.vala#L120) so I could draw the visualizer based on them https://github.com/GeopJr/Tuba/blob/main/src/Widgets/AudioPlayer/Visualizer.vala

[GeopJr]

419518356d8ad151.mp4

[Rafostar]

The audio visualizer requires gstreamer so I had to write everything, including controls. I made them match the GTK ones as Gtk.Video was the default.

ClapperGtkAudio is on my TODO list, since it will allow to reuse the same buttons that can be overlaid on ClapperGtkVideo, but there are lots of things/issues to do, so its not a top priority currently. Whatever you did is "fine" as you get it faster/already this way and it would make Tuba depend on Clapper for yet another thing otherwise.

[Rafostar]

No idea how to make GNOME Builder install the extensions, but outside of it it worked fine.

This seems to be .flatpakref limitation. We tested this with Pipeline/Tubefeeder dev whose app is already updated and when installing app from Flathub this extension is automatically downloaded without issues. When installing from local build, ffmpeg-full is not autodownloaded either.

[Rafostar]

The audio visualizer requires gstreamer so I had to write everything, including controls

Well, you already managed to implement visualizer yourself, but FWIW I recently added a GTK audio-only widget to the ClapperGtk library (link) that works using Clapper player and all the widgets (buttons, seek bar, labels, etc.) that were made for video widget initially.

Not sure whether you would be interested still in it now that you have implemented GStreamer as part of Tuba... just dropping this comment to let you know in case if you ever want to reduce amount of code for audio playback (visualizer) in your app.

[GeopJr]

I like it! I think I might have to write a custom feature to use the level gst plugin (if it's even possible) for the visualizer

[Rafostar]

I might have to write a custom feature to use the level gst plugin (if it's even possible) for the visualizer

No. You do not have to.

1. Use gst_parse_bin_from_description where you add and configure whatever audio elements you need (e.g. level)
2. Set that as player audio-filter
3. Receive messages through player message signal by connecting to message::level with you custom callback

[GeopJr]

Ah, awesome, thanks!

[Rafostar]

Just to make sure, I modified python audio example from here

with:

diff --git a/examples/clapper-gtk/audio/simple/python/example.py b/examples/clapper-gtk/audio/simple/python/example.py
index b477cc9a..91a86af4 100755
--- a/examples/clapper-gtk/audio/simple/python/example.py
+++ b/examples/clapper-gtk/audio/simple/python/example.py
@@ -5,10 +5,14 @@ gi.require_version('Adw', '1')
 gi.require_version('Clapper', '0.0')
 gi.require_version('ClapperGtk', '0.0')
 gi.require_version('Gtk', '4.0')
-from gi.repository import Adw, Clapper, ClapperGtk, Gtk
+gi.require_version('Gst', '1.0')
+from gi.repository import Adw, Clapper, ClapperGtk, Gtk, Gst
 
 Clapper.init(None)
 
+def on_message(player, msg):
+    print('Received level peaks: ' + ','.join(map(str, msg.get_structure().get_value("peak"))))
+
 def on_activate(app):
     # Create our widgets.
     win = Gtk.ApplicationWindow(application=app, title='Clapper Audio', default_width=640, default_height=96)
@@ -19,6 +23,10 @@ def on_activate(app):
     next_btn = ClapperGtk.NextItemButton()
     seek_bar = ClapperGtk.SeekBar()
 
+    audio.props.player.props.audio_filter = Gst.parse_bin_from_description(
+        'scaletempo ! level interval=100000000 post-messages=true', True)
+    audio.props.player.connect('message::level', on_message)
+
     # Add media for playback. First media item in queue will be automatically selected.
     item = Clapper.MediaItem(uri='https://www.soundhelix.com/examples/mp3/SoundHelix-Song-1.mp3')
     audio.props.player.props.queue.add_item(item)

And it worked and started printing level peaks values in terminal:

Received level peaks: -17.43629564952019,-17.457025862782448
Received level peaks: -14.1554506874603,-14.175794335227764
Received level peaks: -14.518214137453334,-14.51587470767307
Received level peaks: -14.359755278080232,-17.554295916920854
Received level peaks: -11.685104099353156,-15.918449156898035
Received level peaks: -10.807955049154698,-18.929448571591827
Received level peaks: -10.153362414320593,-16.64376592338287

So yeah, at least I confirm this is possible to achieve now with few lines of code 😄

[Rafostar]

FWIW I added a simple GTK app example for volume peaks visualizations in Clapper repo, so the above idea/code will not be lost - audio meter example.