fetching package urls from api instead of hardcoded text file

[mAmineChniti]

to avoid limitations that come with files, changing packages and ones that are newly published daily
there's an api that you can just query to get the url and all the information you need about a go package

golang/go#36785 (comment)

https://docs.deps.dev/api/v3alpha/

[RomanVolkov]

@mAmineChniti hey! thanks for sharing, that is actually interesting. I will take a look if I can fully replace with this API

[RomanVolkov]

@mAmineChniti at least at the main page this is exactly what I wanted to find when I researched for this project.
so on the main page: https://deps.dev - there is actual search by substring that returns list of matches.
but they don't use the API directly, the have different endpoint https://deps.dev/_/search/suggest?q=gon&system=GO&kind=PACKAGE

and I checked api docs and still cannot understand which I could use to do the search. Maybe you have an idea?
/GetPackage needs full name
/Query does not return anything without versionKey.version, I tried to send only name and system - no results even for examples.

Of course I can use first endpoint from deps.dev, but I don't think it's a good idea to use undocumented api :)

[RomanVolkov]

okay, I messed with this API more and not sure what actually I can get from it.

• I was not able to implement search by substring (but maybe I miss something)
• I am not sure how I can use e.g. GetPackage to understand if a packages was renamed. E.g., chi router used to be github.com/pressly/chi. If you try to go get it - it will fail

 module declares its path as: github.com/go-chi/chi
 but was required as: github.com/pressly/chi

It would be ideal if I can get implement search with it, that will make a plugin much easier to implement (and expand for python as well), but so far I cannot understand if I can do that with this api

[mAmineChniti]

Maybe you should make github itself your index, search by repo names eliminate forks and non go language repos, that's the only solution i can think about, it's really shit that there's no go packages registry api, npm has one, rust's crates too

[mAmineChniti]

Also users should deal with the renamed packages errors by themselves or if possible you can parse the error and just go get the new one

[RomanVolkov]

@mAmineChniti I actually returned to original approach, but I used deps.dev too
So now I have 2 working checks after I parse proxy.go updates

1. Remove all forks. Most of packages (like 90-95%) are hosted on github and it was easy to understand if publised package is just a fork. If it is - I ignore them and do not add to index
2. Thanks to you @mAmineChniti I'm using deps.dev API to filter all misspelled updated. A lot of cases when packages was published with some type in the name (uppercased vs lowercased). And I managed to filter that with GetPackage endpoint
3. The last one is the most good, but I had to disabled it: https://github.com/RomanVolkov/go.get.nvim/blob/main/index_builder/validations/package.go Ideal that I actually trying to download and install a package into an empty project. If it fails - discard the url. And it works really good. But the problem is that even with batching I hit "not enough space on runner" error. I managed to run it without errors with batch == 200 concurrent installations, but it's very slow. I won't fit into 6h limit for 1 week update of proxy.go. Si I disabled then since I cannot use much bigger runners here as they cost additionally, maybe if this plugin will get some support - then probably yes.
   Another option would be to run it daily, but I think it's too much of commits into the repo just with index updates.

So now index is being update once in a week collecting all the updates first and then filtering them with (1) and (2).

[RomanVolkov]

@mAmineChniti problem with renamed packages that you cannot understand if a packages was moved to another repo. There is no connection. The only way to filter this is option (3)

[mAmineChniti]

yeah 3) is the whole problem with this approach, like having a hardcoded text file and having to parse and check for packages on your own is very unmaintainable is the long run, but there's no other way around it until the official go people create an api for their registry, ayy good luck on this project i'll be using it once it can be installed on lazy.nvim

[RomanVolkov]

@mAmineChniti but you can actually install it already :)
at least I tested with lazy and it worked locally. But I had a local version too, so I would appreciate if you test it and tell me if it worked

[RomanVolkov]

@mAmineChniti in https://github.com/RomanVolkov/go.get.nvim readme there is an information how to call the plugin.

[mAmineChniti]

would be nice if you can include like a lazy.nvim config in the readme

[RomanVolkov]

@mAmineChniti will do👍
I actually want to test other package managers, but I used mostly lazy.nvim with neovim.

[RomanVolkov]

@mAmineChniti
https://github.com/RomanVolkov/go.get.nvim
added

[RomanVolkov]

@mAmineChniti Hey!
I will close this issue, I would say the first version of the plugin is ready to use.
If you have other ideas and suggestions - feel free to open other issues and I would appreciate any feedback here