De-charter Benchmark WG

[mhdawson]

Activity in the benchmarking WG has been low over the last year. After some efforts to try to increase interest/participation and looking for people that would be interested in chairing/leading the group I think it is time to de-charter the WG.

I still believe this is an important area but without critical mass I don't think it makes sense for the WG to continue.

The main thing we'll have to spin down is benchmarking.nodejs.org, diable the related jobs and then archive the repo.

coverage.nodejs.org does still use some of the same machines/infra so we'll need to avoid removing the pieces it needs during the spin down.

[mhdawson]

The job to run the core benchmarks was setup by the Benchmarking WG, but if there is still interest it can just be taken over by core/build as any other CI job.

[mcollina]

This is unfortunate and sad, however we run on volunteers - if there are none, it's good to spin down things. I think the job to run the core benchmark is extremely important and we run it very often, so I think it would be better maintaining it. It has been extremely valuable in the last few years.

[mmarchini]

That's unfortunate, but completely understandable. I was going to ask to join the WG now that I'm back from vacation.

Is there any way we can keep benchmarking.nodejs.org up, even with the WG de-chartered?

[mhdawson]

@mmarchini we can keep it going, the main thing is having people who will help investigate issues/fixup the jobs if they stop working.

[mhdawson]

In terms of the core benchmark is extremely important and we run it very often. I'd agree that we should keep that.

Sounds like we'd like to keep all of the current Benchmarking WG jobs going. One challenge is if we archive the repo is that is where they scripts etc. are maintained. Maybe we'd need to look at migrating them to the core of build repos? Or maybe just leave as is but report issues in the core or build repos?

We'll need some volunteers to help with whatever we decided to keep around. The one for the core benchmark maybe a more manageable as it's smaller/less work but even there its been some trouble getting the cyles to do requested updates.

[mmarchini]

We'll need some volunteers to help with whatever we decided to keep around.

I can help with that. Maybe we should keep the @nodejs/benchmarking team around then, so that others can ping when there are benchmark-related issues?

[mhdawson]

@mmarchini lets find a time to get together and I can do some hand-over/help ramp you up on the jobs and how the infra works to support what's in place.

If there is anybody else who is interested in helping to keep things going please chime in and we'll include you as well.

[Trott]

Assuming adding more work to Build WG isn't going to cause problems, then Build WG seems to be the right place for this. (Maybe mmarchini can join Build?!)

We can keep the repo around and make the Build team admins for it. Team members will be automatically subscribed to repo notifications then, I believe.

Maybe keep the benchmarking team around too. We did that for the Testing WG--dechartered it when the time came, but kept the team around as a group of people to ping about tests. Perhaps the benchmarking team can be expanded to also include people just generally interested in our benchmarks. Then we can finally simplify the entry for who-to-cc-in-the-issue tracker about benchmarks.

[mmarchini]

lets find a time to get together and I can do some hand-over/help ramp you up on the jobs and how the infra works to support what's in place

@mhdawson sounds good. When do you have time to chat?

Maybe mmarchini can join Build?!

I'm on build, but haven't been active for a while :/
Hopefully I'll be able to change that in the mid-future.

We can keep the repo around and make the Build team admins for it. Team members will be automatically subscribed to repo notifications then, I believe.

Maybe keep the benchmarking team around too. We did that for the Testing WG--dechartered it when the time came, but kept the team around as a group of people to ping about tests.

I think keeping the benchmarking team around as well as the repository is a good idea. It would just stop being a WG.

[mhdawson]

@mmarchini would you be available this Friday. Say 9 or 10 EST?

[mmarchini]

I'm available later this Friday (from 12h30 EST to 4:30 EST)

[mhdawson]

@mmarchini, sent invite for 4EST on Friday.

[mmarchini]

@mhdawson can you add me to @nodejs/benchmarking so I receive notifications for benchmark-related issues and PRs? I might also need to join @nodejs/benchmarking-admins to be able to edit the benchmark jobs on Jenkins (I thought I had access but I don't)

[mhdawson]

Added to both.

[mhdawson]

As an FYI, the machine we used to run benchmarks failed so we've disabled the nightly benchmarking runs and don't have any plans to replace them.

[mcollina]

sgtm

The intel machines at NearForm are still running, right?

[mhdawson]

@mcollina yes and they are still being used.

They are being used for the coverage job. We moved that over this week after the original machine failed.

They are also still used for the job that lets people run the micro-benchmarks in the Node.js tree https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/ and there seem to be fairly regular runs.

[mhdawson]

PR to de-charter has landed so closing.

[aduh95]

Now that the repo has been archived, what would be the correct place to move https://github.com/nodejs/benchmarking/blob/master/experimental/benchmarks/community-benchmark/run.sh? It's the file that runs for each benchmark-node-micro-benchmarks job, but the repo is now read-only.