Migrate to Velopack #171

smallketchup82 · 2024-06-28T15:28:24Z

Codependency:

Migrate update framework to Velopack osu#28743

Testing / task lists

Building
- win-x64
- linux-x64
- osx-x64
- osx-arm64
- ios-arm64 (presumed to work)
- android-arm64 (presumed to work)
Uploading / creating releases (https://github.com/smoogipoo/osu/releases)
- win-x64
- linux-x64
- osx-arm64
- osx-x64
Signing
- (@smoogipoo) Update internal deployment repo (https://github.com/ppy/osu-auth-client/pull/63)
- win-x64
- osx-arm64
- android-arm64
Updating to new release (https://github.com/smoogipoo/osu/releases)
- From squirrel build (win-x64
- From velo build (win-x64)
- From velo build (linux-x64)
- From velo build (osx-arm64)
  - You will need to run xattr -c osu\!.app
- From velo build (osx-x64)
  - You will need to run xattr -c osu\!.app

Velopack

I highly recommend reading me and caesay’s discussion in the osu!dev discord to get an idea of what’s going on here.

This is one PR (and the main PR since it contains the largest diff) in a series of PR's to migrate osu!lazer's update framework from Clowd.Squirrel to Velopack, the "next generation" of Clowd.Squirrel.

Why the migration?

Clowd.Squirrel has numerous issues that have long plagued osu!lazer and lazer's development as a whole. Velopack has the following differences compared to Clowd.Squirrel (not sure if they're differences, but it is what it is):

Cross compiling: Velopack allows for compiling a package for a different operating system rather than the current one, saving time.
Automatic shortcut management: Velopack takes care of setting up shortcuts, allowing us to remove our own shortcut creation logic
Release channels: Velopack supports creating numerous release channels, and the methods to do so are very intuitive. For example, It would be entirely possible to provide a new release stream for osu!lazer based on the latest github master commit, tag, etc.
Automatic installers for every platform: Even if the application is packaged portably, say an AppImage on linux, Velopack will facilitate downloading the new AppImage and replacing the old one.
Static executable location: Velopack keeps a folder called current which is a symlink to the most recent release. This allows features like Windows Associations to not require updating each time the game launches.
Easier code signing: Code signing is much easier to do on Velopack than it is on Clowd.Squirrel, which allows us to do less on our end
Automatic downgrading upon switching release channels or if a release gets pulled

All of these would allow for a much simpler osu-deploy, a much cleaner osu!lazer, more opportunities in the future (release channels) and a better user experience for the end user.

So how's it going to work?

I'm going to be basing my osu-deploy branch off of caesay’s branch, while keeping it in line with recent osu-deploy updates and updates to velopack. The same can be said for my osu!lazer branch, though I unintentionally reimplemented caesay's branch instead of basing it on his branch.

Velopack is more or less a drop in replacement for Clowd.Squirrel, so all that really needs to be done is to remove lines and change existing lines to use Velopack instead. Velopack will automatically manage reading from Squirrel's RELEASE files and migrating clients onto Velopack, so there is no work needed to be done on that front.

Both me and caesay want to discuss things before doing things, so as to not waste effort. Which is why this PR, while it is in it's draft phase, will be the main place for discussion on Velopack migration, design choices, considerations, etc.

Differences from caesay's branch

Decided to not change the GithubRelease and GithubObject models, and also not add a GithubReleaseHelper
- Mainly did this because I believe it's out of scope of this PR, and would like to keep the file change count low. This movement of logic is better suited for a future PR
Decided not to add a Log class
- For the same reasons as the last. This is better suited for a future PR/refactor
Removed unused (probably meant for the future) functions
Shuffled around the logic in caesay's branch to try and reduce the amount of if statements needed
Updated old stuff to work with the latest version of Velopack and osu-deploy (e.g. appdir handling)'

Final notes

Not much to say here. Maintainers can edit the branch as needed. @caesay if you want to make changes to my branch, I'm okay with either a PR on my fork or a git patch. This should close #125 as a side effect (as vpk pack will output a nupkg, removing the need for nuget.exe). And this will make all open PR's obsolete.

smallketchup82 · 2024-06-30T03:33:30Z

I've updated the PR description with necessary information. I'm open for any discussions on this.

To keep things organized, here are some quick links to relevant branches:
osu!lazer | osu-deploy | caesay's osu-deploy branch

To get things started, I'd like for some feedback on my osu-deploy branch.

- Add github upload ability - More work done on cross compiling - Add vpk tool locally so that it gets installed via a dotnet restore - Remove Clowd.Squirrel and NuGet.CommandLine as dependencies

Velopack automatically manages and optimizes the number and size of deltas

- Fix bug where commands would fail to run because "dotnet vpk" isn't an executable - Fix upstream bug where linux packaging would break due to the appDir command - Remove Velopack as a dependency because it isn't needed

caesay · 2024-07-19T08:58:47Z

Let me know if there's anything I can do to help move this along

smallketchup82 · 2024-07-19T18:35:44Z

Let me know if there's anything I can do to help move this along

I'm waiting on review. If you'd like, you can give this a review, though I'm not sure how much of a help that would be since the core team would likely want a review of their own.

It'd be nice to know where this feature stands priority-wise. If we're looking at a couple months until it's given attention, I'll probably go ahead and work on something else in the meantime.

peppy · 2024-07-20T04:34:50Z

To me this is an up-next feature. I've discussed with smoogi getting it attention sooner rather than later.

So when I converted this to use local tools instead of global tools, it turns out that I forgot to account for the *working directory* of the commands. It was being set to the osu!.Desktop solution folder, rather than osu-deploy, resulting in an error that the vpk tool couldnt be found. This fixes that, and since the commands process absolute paths anyways, it doesnt really change anything.

This version is a stable version, unlike the pre-release we were on beforehand. It also includes all of the changes I asked for, making it a pretty seamless upgrade. I've tested a deploy using it, and it works.

smoogipoo · 2024-08-07T07:36:08Z

I'm in the process of refactoring this entire PR because it's unreadable as it is, please hold off on pushing additional fixes for now.

(it's also broken on macOS, but again, this is something I'm fixing/have fixed)

smallketchup82 · 2024-08-28T22:47:05Z

@smoogipoo I'm going to be starting my last year of High School on September 3rd, so I'll likely be inactive from the 3rd to December (winter break). I will not be checking Discord, and I probably won't have the time to make large changes (I'll probably end up having to spread work across weeks & months rather than days).

Felt like letting you know cause I do have a few PR's open (velopack & normalization), and you seem to be the one working through most of them. Since I won't be as available starting a week from now, It'd be nice if we could try to get some work done on my PR's (or just this one) before then.

Let me know what you'd like to do here. I'd be okay with handing my branches over if it comes to it.

"mac" isn't supported for `dotnet build`, and looks like vpk also uses/can use "osx".

In particular, the `--icon` argument requires a specific icon type. We already have our own prebuilt bundle with the icon _and_ other things like the plist.

smoogipoo · 2024-08-30T16:01:38Z

Sorry for taking so long, I've been jumping around several bigger tasks and this one got delayed as a result.

I'm going to take a different approach, which is to just PR changes to your fork. See smallketchup82#1 + smallketchup82#2.

smallketchup82 · 2024-08-30T19:59:02Z

No problem, I would've been fine with waiting longer but I'm getting a little tight on schedule. I'm going to try and get this finished ASAP while I still can.

PRing to my fork works, I'll give the PR's a review and test them myself. But just to make sure, are the currently open PR's on my fork all there is, or do you still have more work to do on refactoring?

smoogipoo · 2024-08-31T08:08:40Z

I suppose we can rename the portable package/installer post-upload since it's not used elsewhere. Will try going down that route.

smoogipoo · 2024-08-31T12:18:32Z

I think I'm done now, with all the latest changes PR'd to the fork. Will need to wait for a velopack release containing velopack/velopack#211, and update the signing on our deployment repo, and then this should be good to go.

For testing, my fork has three releases all hooked up: https://github.com/smoogipoo/osu/releases

A squirrel release from the latest master, used to test Squirrel->Velopack update path.
Two Velopack releases, used to test Velopack->Velopack update path.

Will likely update the signing and do a final test deploy tomorrow.

smallketchup82 · 2024-08-31T14:10:21Z

@smoogipoo If I'm not mistaken, can't we just use this for the time being?

https://github.com/velopack/velopack/releases/tag/0.0.598-g933b2ab

caesay · 2024-08-31T14:14:42Z

0.0.598-g933b2ab has the fix from smoogipoo in it already, I just released it.

smoogipoo · 2024-08-31T14:22:22Z

Oh awesome, thanks @caesay!

This is ready to go. We likely won't do further testing/release until Monday at the earliest due to the complexities involved, but everything should now be in place.

smoogipoo · 2024-08-31T14:32:53Z

@peppy You might want to use fresh eyes to review this one. I struggled with the original code so I took the liberty of separating everything out into Builders and Uploaders.

I've tried to keep the core logic familiar for now, so it's not entirely clean - there's still references to statics existing in Program. It's just separating the core logic out into per-platform tasks.

I'd suggest refactoring further if required as a separate PR.

peppy · 2024-09-01T13:39:41Z

@smoogipoo is it save to assume the OP checklist is out of date? it seems like signing is handled in https://github.com/ppy/osu-auth-client/pull/63 and osx-x64 is building?

smoogipoo · 2024-09-01T13:44:50Z

Checklist is up to date - I haven't tested signing at all, and I also haven't tested updating on osx-x64 though I presume it works given arm64 does.

The releases on https://github.com/smoogipoo/osu/releases have x64 binaries ready to be tested, I just don't have an x64 machine around me atm.

peppy · 2024-09-01T13:50:35Z

Okay awesome, I'll test those.

peppy · 2024-09-04T07:17:59Z

Builders/WindowsBuilder.cs

+                bool wineRcEditCommand =
+                    Program.RunCommand("wine", $"\"{Path.GetFullPath("tools/rcedit-x64.exe")}\""
+                                               + $" \"{Path.Combine(Program.StagingPath, "osu!.exe")}\""
+                                               + $" --set-icon \"{IconPath}\"",
+                        exitOnFail: false);


Is this something specific to your operating environment @smoogipoo, or something github actions related? I know for a fact that I have wine installed nowhere.

This is how it is in master, dating back to 2018 or so. I don't remember the reason why this exists in the first place, but It looks like .NET is handling the app icon fine via ApplicationIcon at least when compiled on Windows (which is what our runners do anyway). So... we can maybe remove this, but as I mentioned above I intended to keep the code familiar and not introduce too many changes.

Oh okay. I checked and it looked like it didn't exist but I guess I missed it.

Maybe I'm going crazy. I definitely thought I saw it there but I can't find it, and it looks like it was added in this PR (the Wine part)...

I guess it's not a big issue either way? Though I still want to explore removing all of this altogether as a followup?

maybe @smallketchup82 can answer this then

Documenting here for now because I was about to remove this... It appears that ApplicationIcon sets the icon correctly for the shortcut, taskbar, and alt-tab, but the window titlebar loses its icon.

Which is weird because we have SDL code to handle that... https://github.com/ppy/osu-framework/blob/e0e945672d41c395c1977d273220cb17f9d52e2a/osu.Framework/Platform/Windows/SDL2WindowsWindow.cs#L293

Needs more investigation.

maybe @smallketchup82 can answer this then

Yes I added this in. Meant to be a sort of "fallback" method when compiling for windows on a linux system. There's no equivalent unix binary for that tool (at least to my knowledge), so the best way to run it is via wine (if the user has it installed). If the user doesn't have wine installed, it'll fail gracefully and move on.

Seems to work perfectly, and removing it more or less breaks cross compilation (icon's don't get set). Best to keep it in so that the option of using it is available

Velopack now has a cross-platform resource editor written in C#, as it needs to set icons on it's own Update.exe even when on Linux. I guess you could steal this and make your own small utility instead of using wine+rcedit. https://github.com/velopack/velopack/blob/develop/src/vpk/Velopack.Packaging.Windows/ResourceEdit.cs

peppy · 2024-09-04T07:30:40Z

Merging this based on local testing and code review to reduce moving pieces. Will test signing as the final step.

smallketchup82 · 2024-09-06T12:41:19Z

Thanks for all the help!

smallketchup82 added 2 commits June 26, 2024 15:25

initial implementation

e9c11a0

Commit whatever I already have so that I can make the PR

01ca7c4

smallketchup82 added 4 commits June 30, 2024 03:26

Get things stable

243bb7d

Overhaul

b4f265d

- Add github upload ability - More work done on cross compiling - Add vpk tool locally so that it gets installed via a dotnet restore - Remove Clowd.Squirrel and NuGet.CommandLine as dependencies

Run jetbrains formatter

1453813

Remove keep deltas variable

4549aa4

Velopack automatically manages and optimizes the number and size of deltas

smallketchup82 mentioned this pull request Jul 4, 2024

Migrate update framework to Velopack ppy/osu#28743

Merged

5 tasks

Bug fixes

402bd4e

- Fix bug where commands would fail to run because "dotnet vpk" isn't an executable - Fix upstream bug where linux packaging would break due to the appDir command - Remove Velopack as a dependency because it isn't needed

smallketchup82 marked this pull request as ready for review July 4, 2024 22:44

smallketchup82 added 2 commits August 7, 2024 03:32

Bump velopack version

1555f2d

This version is a stable version, unlike the pre-release we were on beforehand. It also includes all of the changes I asked for, making it a pretty seamless upgrade. I've tested a deploy using it, and it works.

peppy assigned smoogipoo Aug 7, 2024

smoogipoo added 4 commits August 30, 2024 21:26

Apply auto refactoring

5b86d4b

Split into individual builders

0e07b2c

Use correct OS name for macOS

a52a207

"mac" isn't supported for `dotnet build`, and looks like vpk also uses/can use "osx".

Use custom app bundle

6a2cf44

In particular, the `--icon` argument requires a specific icon type. We already have our own prebuilt bundle with the icon _and_ other things like the plist.

smoogipoo added 6 commits August 31, 2024 01:21

Restore tools before build

604916a

Always output stopwatch time

b377ef9

Add signEntitlements

6d43dc2

Don't generate package for now

d67f460

Fully specify option names

f3d2c88

Add keychain option

acfae8d

Merge pull request #5 from smoogipoo/no-portable-win

82fce38

smoogipoo added 4 commits August 31, 2024 20:14

Rename assets to match existing releases

5cef162

Don't publish ios/android builds if requested not to

79494ab

Fix windows exe name

fab6a45

Add --packTitle to pack command

7096183

smallketchup82 added 3 commits August 31, 2024 10:05

Merge pull request #7 from smoogipoo/conditional-upload-ios-android

8256034

Merge pull request #8 from smoogipoo/add-pack-title

2cd4e0a

Merge pull request #6 from smoogipoo/rename-assets

2076306

Update Velopack package

6829172

smoogipoo requested a review from peppy August 31, 2024 14:22

smoogipoo and others added 4 commits August 31, 2024 23:59

Fix Windows installer icon

c1bebab

Merge pull request #9 from smoogipoo/fix-windows-setup-icon

4e7b398

Restore tools in correct directory

c4092ca

Add back removed line + refactoring

ae39a9c

Add missing dictionary and abbreviation entries

16ad94b

peppy reviewed Sep 4, 2024

View reviewed changes

peppy approved these changes Sep 4, 2024

View reviewed changes

peppy merged commit ca43ff3 into ppy:master Sep 4, 2024
1 of 2 checks passed

smallketchup82 deleted the velopack branch September 6, 2024 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate to Velopack #171

Migrate to Velopack #171

smallketchup82 commented Jun 28, 2024 •

edited by peppy

Loading

smallketchup82 commented Jun 30, 2024 •

edited

Loading

caesay commented Jul 19, 2024

smallketchup82 commented Jul 19, 2024

peppy commented Jul 20, 2024

smoogipoo commented Aug 7, 2024

smallketchup82 commented Aug 28, 2024 •

edited

Loading

smoogipoo commented Aug 30, 2024

smallketchup82 commented Aug 30, 2024 •

edited

Loading

smoogipoo commented Aug 31, 2024

smoogipoo commented Aug 31, 2024 •

edited

Loading

smallketchup82 commented Aug 31, 2024

caesay commented Aug 31, 2024

smoogipoo commented Aug 31, 2024

smoogipoo commented Aug 31, 2024

peppy commented Sep 1, 2024

smoogipoo commented Sep 1, 2024 •

edited

Loading

peppy commented Sep 1, 2024

peppy Sep 4, 2024

smoogipoo Sep 4, 2024

peppy Sep 4, 2024

smoogipoo Sep 4, 2024 •

edited

Loading

peppy Sep 4, 2024

smoogipoo Sep 4, 2024 •

edited

Loading

smallketchup82 Sep 4, 2024 •

edited

Loading

caesay Sep 4, 2024 •

edited

Loading

peppy commented Sep 4, 2024

smallketchup82 commented Sep 6, 2024

Migrate to Velopack #171

Migrate to Velopack #171

Conversation

smallketchup82 commented Jun 28, 2024 • edited by peppy Loading

Testing / task lists

Velopack

Why the migration?

So how's it going to work?

Differences from caesay's branch

Final notes

smallketchup82 commented Jun 30, 2024 • edited Loading

caesay commented Jul 19, 2024

smallketchup82 commented Jul 19, 2024

peppy commented Jul 20, 2024

smoogipoo commented Aug 7, 2024

smallketchup82 commented Aug 28, 2024 • edited Loading

smoogipoo commented Aug 30, 2024

smallketchup82 commented Aug 30, 2024 • edited Loading

smoogipoo commented Aug 31, 2024

smoogipoo commented Aug 31, 2024 • edited Loading

smallketchup82 commented Aug 31, 2024

caesay commented Aug 31, 2024

smoogipoo commented Aug 31, 2024

smoogipoo commented Aug 31, 2024

peppy commented Sep 1, 2024

smoogipoo commented Sep 1, 2024 • edited Loading

peppy commented Sep 1, 2024

peppy Sep 4, 2024

Choose a reason for hiding this comment

smoogipoo Sep 4, 2024

Choose a reason for hiding this comment

peppy Sep 4, 2024

Choose a reason for hiding this comment

smoogipoo Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

peppy Sep 4, 2024

Choose a reason for hiding this comment

smoogipoo Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

smallketchup82 Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

caesay Sep 4, 2024 • edited Loading

Choose a reason for hiding this comment

peppy commented Sep 4, 2024

smallketchup82 commented Sep 6, 2024

smallketchup82 commented Jun 28, 2024 •

edited by peppy

Loading

smallketchup82 commented Jun 30, 2024 •

edited

Loading

smallketchup82 commented Aug 28, 2024 •

edited

Loading

smallketchup82 commented Aug 30, 2024 •

edited

Loading

smoogipoo commented Aug 31, 2024 •

edited

Loading

smoogipoo commented Sep 1, 2024 •

edited

Loading

smoogipoo Sep 4, 2024 •

edited

Loading

smoogipoo Sep 4, 2024 •

edited

Loading

smallketchup82 Sep 4, 2024 •

edited

Loading

caesay Sep 4, 2024 •

edited

Loading