[WIP] vscode-extensions: support multiple extension registries

[ShamrockLee]

Motivation for this change

According to the Microsoft Visual Studio Marketplace Term of Use, VSCodium and editors other than the officially-built Visual Studio Code are prohibited from using "Marketplace Offerings" including extensions. These editors thus use the Open VSX Registry to provide extension searching/installation features instead of the Visual Studio Marketplace .

If applied, extensions will be downloaded from the Open VSX Registry by default to make them legally available to users of open-source vscode builds and other derived editors.

This PR provides a framework to supply extensions from different registries. Four registies, vscode-marketplace, openvsx, upstream and standalone, will be added.

Two nix-prefetch packages, nix-prefetch-openvsx and nix-prefetch-vscode-mktplc are included to make pre-fetching extensions. The output will be in JSON format similar to nix-prefetch-github.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[ShamrockLee]

Modified nix-prefetch-openvsx and nix-prefetch-vscode-mktplc to correctly delete the temporary directory.

The original way to handle exit in the update_installed_exts.sh only handles SIGINT (Ctrl-c) and successful exit, and will also delete the temporary directory used by another instance of this program.

[ShamrockLee]

@SuperSandro2000
It would be preferable to merge #113280 first and I'll then integrate with it.

1. update_installed_exts.sh

   • misc/vscode-extensions: improve scripts for updating VSC extensions #113280 changes the way argument lists are processed.
   • This pr changes the way extensions are fetched.
   • The script needs integration.

2. Documentation

   • misc/vscode-extensions: improve scripts for updating VSC extensions #113280 adds the documentation about VSCode and extensions.
   • The changes in this PR needs to be documented, and the sha256 values of the examples need to be changed.
   • Extra stuff to document will include:
     • Sources of extensions (Open VSX Registry / official Marketplace)
     • Two new prefetch scripts for extensions (nix-prefetch-openvsx and nix-prefetch-vscode-mktplc)

3. Backward compatibility

   • misc/vscode-extensions: improve scripts for updating VSC extensions #113280 is totally backward compatible backward incompatible on the update script, and user-defined extensions won't be influenced.
   • This PR changes the default source of the extensions, and may affect the sha256 values of the extensions locally fetched by users.

[SuperSandro2000]

It would be preferable to merge #113280 first and I'll then integrate with it.

Thats probably a good idea but I don't know the vscode package system that well to review the PRs well.

[ShamrockLee]

Exposed useMSMktplc for future module making, with similar functionality to meta.license.free.

[ShamrockLee]

Here's the code implementing the serviceUrl and itemUrl adapter in the Open VSX project:

https://github.com/eclipse/openvsx/tree/f653161c7bd9b08ae80ba24df758839d72bc0fe0/server/src/main/java/org/eclipse/openvsx/adapter

The corresponding urls are:

"extensionsGallery": { "serviceUrl": "https://open-vsx.org/vscode/gallery", "itemUrl": "https://open-vsx.org/vscode/item" }

They may help us figure out a way to query the extension list of VSCode Marketplace.

[ShamrockLee]

I originally plan to extract extension identifiers from the index.json fetched from the obsolete cacheUrl (https://vscode.blob.core.windows.net/gallery/index) so that we can have 2000+ new extensions from the Marketplace.
Now I think we had better drop the file (index) and prevent any of the derived work from entering the source tree considering the EULA of the VSCode Marketplace.

BTW, I'm working on some fixes related to my research work, and is running out of time recently. It would take a while before I could port the custom build process here from vscode-extension. (The extensions that only go through the standard build process should now be usable.)

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/in-tree-or-out-of-tree/18389/1

[ShamrockLee]

I just encountered something called extension packs, which aren't managed properly by the current vscode-extensions framework.

It's quite easy to get the information -- by fetching the attribute extensionPack from package.json when bootstrapping / updating registry-reference-list.json, but we need to find a way to inject those dependent extensions and their dependent extensions ... into the VSCode / Codium.

[schuelermine]

hey @ShamrockLee are you still working on this? I recently thought of the same kind of thing so I opened #197293, and I’m wondering if I should close in favor of this PR.

[ShamrockLee]

This one is still relevant, and it should already work on extensions that doesn't require custom build process.

We could also handle the extensionPack that comes along an extension with the new framework. It's rather easy to implement.

There are some problems, through:

• I wonder if the overlay-style build process abstraction in this PR would be over-enginneering.

• The custom build process (for extensions which needs more than unpack-and-installing to get built) needs to be ported manually to the new framework. It could be shared between registries once the name issues mentioned below could be resolved.

• Extensions are identified with their publisher.name, which is generally the same accross registries. The package attribute name of each extension is usually identical to its publisher.name. However, the exceptions are not neglectable. One example of the name-nad-attribute mismatch is the package identified as "bbenoist.Nix" being renamed from vscode-extensions.bbenoist.Nix to vscode-extensions.bbenoist.nix. Should we just insist that the attribute be identical to publisher.name in this new framework? The naming difference between registries should be a rather mild problem, as custom build process can be mapped to different names.

[schuelermine]

Should we just insist that the attribute be identical to publisher.name in this new framework?

I think that sounds reasonable. Couldn’t any other things be done via the aliases in the current system anyway?

[ShamrockLee]

The attribute renaming was made in commit 9ee8bd1 . It casted the letter to lower case to "keep the naming convention uniform".

[schuelermine]

that doesn’t sound reasonable. IMO if the source is inconsistent it’d be inconsistent to make it consistent

[ShamrockLee]

@schuelermine I didn't even realize that .vsix files are now provided per-platform before your PR. Do you know how the platform-specific packages differ from the ones downloaded without specifying the platform? BTW, is there a way to download .vsix files platform-wise from VSCode Marketplace?

[schuelermine]

@ShamrockLee
No, I don't know anything about it. I just copied the platform distinction from the API reference for Open VSX and I reused the arch property from the marketplace code, assuming it refers to the same thing.

[ShamrockLee]

On 02.08.2023 15:44, Mateusz Franik wrote: Are there any updates on this?

Sorry for disappearing for so long. Here's the things done: 1. `registry-references-list.json` from OpenVSX can be generated programmatically using `nix-prefetch-openvsx`. 2. `registry-references-list.json` from VSCode Marketplace are generated manually, but is probably outdated. 3. All the extensions from `registry-references-list.json` that doesn't need special build process can be built. Todo: 1. Unite the naming convention. Quite some extension attributes are changed to lower case instead of following the "extension publisher" and "extension name" from the upstream. I personally think it's a good idea to rename them back. 2. Port all the custom build process to to the new framework. This requires a lot of manual work against a moving target (the current extension set), and is the main blocker of this PR. 3. The old VSCode marketplace asset URLs cannot fetch some new extensions, and the relevant URLs seem to come with certain filters that blocks direct `curl`s. This affects not only this PR and the `nix-prefetch-vscode-marketplace` prefetcher, but also the current extension set (built with `vscode-utils.buildVscodeMarketplaceExtension`). We would need to solve this with VSCode-like User Agent, or even custom headers. This require inspecting against the VSCode OSS source code. 4. Automate production of the `registry-references-list.json` from the current extension set. This requires the hash-conversion builtin function (NixOS/nix#7708). With such builtin function, we could also convert SHA256 hash fetched from OpenVSX without actually downloading the VSIX file and compute the hash. 5. Simplify the overlay structure. The one in this PR seems over-engineered.

…

-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because you were mentioned.Message ID: ***@***.***> Links: ------ [1] #121583 (comment) [2] https://github.com/notifications/unsubscribe-auth/AKQF2M2HUAT2M66WHRV7W3DXTJKSLANCNFSM44AP6DBA