remove DelimitedFiles from being an stdlib

[KristofferC]

Requires JuliaLang/Pkg.jl#3097 and JuliaRegistries/General#61457

Fixes #44663

[ViralBShah]

@KristofferC I think you mentioned somewhere that there were some issues with Compat. Perhaps good to capture those here.

[KristofferC]

Yes,

So as it is right now, there are a bunch of packages that depend on standard libraries. The compatibility requirements for stdlib dependencies are implicit in the julia compatibility. Now, a stdlib gets moved out to become a standard package (let's call this an ex-stdlib). It will be versioned like a normal package and people with put compat on it like a normal package. There are a few questions that arises now:

• When a package cannot be installed because there is a conflict with the compatibility in an ex-stdlib, Pkg will instead pick an old version of the package that has the ex-stdlib dependency but without any compatibility constrained (which exist from the time the ex-stdlib was an stdlib). Installing very old pacakges is unlikely what you want as a user.
• When a package adds compat to the ex-stdlib what will happen to older Julias where the package is still and stdlib? Will they be able to install the package?
• Is it important what version the ex-stdlib has when it is moved out. Should it start at 1.9 (since Julia is at 1.9).

I think I have to play around with this a bit locally and create "fake packages" that set up these scenarios to conclusively say what is best to do here.

[carstenbauer]

If I may ask, what's the reasoning / motivation for dropping DelimitedFiles as a stdlib? I might have missed it but haven't seen a discussion about this anywhere.

[ViralBShah]

This is being planned not just DelimitedFiles, but also Statistics and the sparse ecosystem. In my opinion, there are several benefits:

1. Allow broader participation in these packages from contributors
2. Faster bugfixing and upgrades without having to wait for whole Julia release cycles
3. Make them less special and allow for alternatives to evolve, or become more flexible (e.g. we want to support many new sparse data types and solvers in a first class way)
4. Consolidate capabilities in certain ecosystems (e.g. basic stats is spread out across too many packages like Statistics.jl, StatsBase.jl etc., and it is complex for new users to navigate - not to mention it is difficult to maintain)

Various things are being planned to make this non-breaking, and more are discovered as we go forward:

1. The spun off packages will be 1.x.y version numbers so that there is no API breakage.
2. With Project and Manifest files, the stdlibs are already dependencies
3. Updating Pkg and related code to make this smooth.

[brenhinkeller]

2 is very valid, and 4 would be awesome!

3 is interesting, in that on one hand that's almost certainly true, but on the other hand it can be nice to have Base and Stdlibs as an "api" of sorts that other packages can extend (i.e., you should import and extend Statistics.mean for your custom fancy type, don't just roll your own conflicting mean) and with more things spun out of stdlibs it'll be less clear what to extend

I still have a knee-jerk reaction ("surely this will end up breaking someone's workflow anyways" "ok, so they'll still get checked in pkgeval, but what if something breaks them anyways, there won't be as much pressure to fix downstream", etc.), but that's probably paranoid. All in all it's probably (?) worth it as long as it really really isn't breaking and everyone's Project.tomls will get PRd with appropriate compat bounds and such?

[juliohm]

but on the other hand it can be nice to have Base and Stdlibs as an "api" of sorts that other packages can extend (i.e., you should import and extend Statistics.mean for your custom fancy type, don't just roll your own conflicting mean)

I don't think that these need to live in Base or as stdlib. They are much better suited for a StatsBase.jl or StatsCore.jl or StatsAPI.jl package that is developed freely by the statistics community in Julia without language release cycles and related concerns.

[KristofferC]

We want to get rid of SuiteSparse test dependency on DelimitedFiles.

Regarding the compat problem, I think things should be ok if we retroactively add compat to an ex-stdlib, duplicating that of the Julia version, and setting the version of the ex-stdlib to that of the Julia version when it moves out. So DelimitedFiles would get version 1.9.0 in this case.

Here is a commit that adds the compat to DelimitedFiles as an example: JuliaRegistries/General@969344d.

Made with https://github.com/KristofferC/StdlibRegistryCompatUpdater

[ViralBShah]

SuiteSparse test dependency on DelimitedFiles is fixed in #45972

[ViralBShah]

Wondering if we can get this in before the 1.9 feature freeze.

[maleadt]

@nanosoldier runtests(ALL, configuration=(registry="kc/consistent_spacing", ), vs = ":master")

[nanosoldier]

Your package evaluation job has completed - possible new issues were detected. A full report can be found here.

[maleadt]

No new errors about uninstallable packages, so that looks good?

[KristofferC]

Yep!

[KristofferC]

I want to run the Pkg tests using the JuliaRegistries/General#66039 branch of the registry. If that passes, I think we are good to go.