Benchmarking: where should it live?

[brainstorm]

We'd like to work on benchmarking our baseline: Noodles.

Shall we just fork Noodles, work on some benchmarks that illustrate different access patterns/applications and then PR it upstream? Or a separate repo altogether?

/cc @multimeric @mmalenic @E-Allie

[multimeric]

Copying from Slack:

I think my preference is to put the benchmarking all in the same repo. At least the way I would write it, we would have some kind of BioBenchmark Rust trait that we impl for each format, and that provides a benchmark method that we can run. But organizing it neatly in that way wouldn't work across two repos.

[brainstorm]

Understood, two things:

1. After our call you made me reflect about ownership of the benchmarks. I think that your approach on keeping our benchmarking to ourselves (this repo) instead of PR/upstreaming it to Noodles right away might be better, you convinced me.
2. I'm not too sure about the special Rust trait though. I'd go with what criterion or hyperfine crates docs suggest. I've learned that fighting against crate assumptions about project structure can backfire :-S

[multimeric]

I've had a small look at criterion, and I agree that we should work within its API, but specifically I think we should leverage the benchmark_group concept here to avoid duplicating code. For example if we had a BioFormat trait, we could then make a Vec<Box<dyn BioFormat>> and iterate over them in the benchmark_group, calling format.save(), format.load() etc, using polymorphism to avoid having to rewrite a new benchmark for each format.

[brainstorm]

Neat, that does seem to make sense. @E-Allie, thoughts?

[E-Allie]

I agree that it should all live in the same (this) repo. We could always pin/submodule some revision of Noodles to our repo to demonstrate functionality upon it, but that does not mean it needs to be upstreamed in the end.

Some self-owned BioFormat trait would also make the most sense in this context, as it could then be impl'd on externally owned types as well.