[RFC] New release schedule

[vondele]

Traditionally new releases have been created once the master branch was clearly 50Elo stronger than the last release.

https://github.com/glinscott/fishtest/wiki/Regression-Tests#fishtest-links

This made sense in the past, for example, because releases needed some additional testing to guarantee correctness, and to avoid flooding 'the market' with small updates.

However, things have changed, and it is time to revisit our release model. For the following reasons:

• Master is almost always in very good shape because our CI has become rather reliable.
• It is relatively straightforward to do a release, the process could be further automated
• The majority of our millions of users rely on released versions. There is a small community of enthusiasts that will use a dev version, or build their own, but, as for most software, people rely on released versions.
• As the engine becomes stronger, Elo gains must decrease, as draw rates in self play go up. Releases would become very rare.
• Long delays between releases are poor public relations, other projects have taken advantage of this to profile themselves.
• If we participate in tournaments, we don't send releases but devs versions. We feel releases are out-dated.

I propose we change the release model in the following way:

• We switch to normalized Elo as obtained with the pentanomial statistics as the base model. The following doc provides a good intro. This has the advantage that this:
  • remains meaningful as the drawrate increases
  • is slightly less sensitive to the opening book
• We make a major release (e.g. SF13 -> SF14) once we achieved 100 nElo (normalized Elo)
• We make minor releases (e.g. SF13.1 SF13.2 SF13.N ) whenever we fix a significant bug, or we are roughly 20 nElo better than the previous major/minor release.

This has the advantages that

• We will have major releases with roughly the same time between major releases (~1yr) as we had in the past (100 nElo was roughly what old released had)
• We have a minor releases every quarter or so.
• We have a way to propagate critical bug fixes to the users.

Comments, alternatives ?

[scchess]

Stockfish has been quite stable and is always being tested every second by a dozen of people all over the world, so it probably won't come with any significant bug.

[vondele]

Over the years there have been a number of small bugs, like extremely rare crashes, or wrong mate announcements. A 'recent' example would be 4e2bb8f

[osteslag]

Your arguments for more frequent releases make sense.

I’m not necessarily advocating this, but I thought it’s worth at least having the discussion:

An alternative to the proposed semantic versioning could be a sequential calendar-based scheme, where the versions would become SF2021.1 (==SF13), SF2021.2, SF2021.3, …, SF2022.1, … which communicates a point in time instead of an implied gain in strength and/or a fix.

You’d lose is the implied strength increase, yes, but this could also be liberating, because we wouldn’t see large minor release numbers, e.g., SF13.12 or, if you’re accumulating strength between the major versions, you’d perhaps release SF14 with only marginal gain over SF13.12, just because nElo gain between SF13 and SF14 had accumulated to more than 100, if you are so inclined.

There’s a discussion on sequential calendar-based versioning (in relation to iOS apps) in this podcast episode.

[Sopel97]

I agree with all of it. The pipeline for generating releases needs some upgrading though as you pointed out. I started working on a better release pipeline for windows (mingw) that would actually produce quality binaries for all archs with minimal effort. It could be adapted to linux with little effort too I think.

[vondele]

Yes, I've seen the PR, that would allows us to do PGO builds for most supported archs. I think this should work on linux as well. I'll try this out in the future.

[vondele]

Someone with the knowledge should probably pickup the suggestion here #2958 (comment) on how to get to a CI/CD process that builds the binaries automatically.

[manyoso]

"If we participate in tournaments, we don't send releases but devs versions. We feel releases are out-dated."

Does this mean following this new release model that this would be fixed? ie., SF would send only releases to participate in tournaments and not dev versions?

[vondele]

probably not, even with the new scheme releases will always be a little bit behind. The difference would just be smaller

[manyoso]

Assuming you have your release scripts in tip-top shape you could always add another minor number for every patch shrug

[Lolligerhans]

For the casual users, who are analyzing their games or preparing openings (and who are not aware of abrok), the latest 25 elo will hardly be any different.

Bundling the engine with an updating script/demon/loader would in my experience be more convenient for users. The users (potentially) benefitting from more frequent releases are those who do not want to check the progress themselves, they just want it to work.

[vondele]

an updater seems like it would be part of a distribution/package manager or a GUI. I could imagine some Linux distribution or other package managers could be picking up releases quite easily.

[MichaelB7]

I'm all for this, my only suggestion is perhaps make it every 4 months , so call it a triannual schedule or 3 times a year. Every 3 months just feels a little short for me and also , releases do require a little additional work and by going from 4 release annually to 3 releases, you just reduced that burden by 25%. Anyway , just a suggestion, I will support whatever decision is made.

[vondele]

20 nElo as suggested above could be varying in time.

[xoto10]

If the problem is less frequent versions as elo gains get harder, I would just make the releases time dependent, i.e. once a year.
Perhaps the minor releases could be slightly more flexible, e.g. "probably" between 3 and 4 months, or even just unspecified, to allow the maintainers to adapt their release to circumstances.