feat: optimize indices

[eddyxu]

Allow users to speicfy how many delta indices to be merged

[westonpace]

A few questions to help me understand how this operation works

[westonpace]

Again, could maybe use Vec::from_iter here?

[eddyxu]

When I change to

    let indices = stream::iter(old_indices.iter())
        .zip(repeat((dataset.clone(), col_name.clone())))
        .map(|(meta, (ds, col_name))| async move {
            ds.open_generic_index(&col_name, &meta.uuid.to_string())
                .await
        })
        .buffered(10)
        .try_collect::<Vec<_>>()
        .await?;

it complaints

error: higher-ranked lifetime error
   --> lance/src/index.rs:300:5
    |
300 |     #[instrument(skip_all)]
    |     ^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: could not prove `Pin<Box<{async block@lance/src/index.rs:300:5: 300:28}>>: CoerceUnsized<Pin<Box<(dyn futures::Future<Output = Result<(), lance_core::Error>> + std::marker::Send + 'i)>>>`
    = note: this error originates in the attribute macro `instrument` (in Nightly builds, run with -Z macro-backtrace for more info)

error: could not compile `lance` (lib) due to previous error

[westonpace]

Ah, the return of the dreaded rust-lang/rust#102211

Don't worry too much about it but I I've had luck with something like this...

    let indices = stream::iter(old_indices.iter())
        .zip(repeat((dataset.clone(), col_name.clone())))
        .map(|(meta, (ds, col_name))| async move {
            ds.open_generic_index(&col_name, &meta.uuid.to_string())
                .await
        })
        .collect::<Vec<_>>();
   let indices = stream::iter(indices)
        .buffered(10)
        .try_collect::<Vec<_>>()
        .await?;

...or you can just remove the call to buffered (that's the method that usually introduces the bogus error).

[westonpace]

I think we need to explain to the user why they would change this parameter.

[eddyxu]

Updated the doc

[westonpace]

Don't worry too much. We can tweak this if/when the parameter becomes public. I'm guessing that there is some kind of cost-to-compute/accuracy-of-index tradeoff here? Or is it a cost-to-compute/cost-to-search tradeoff?

In other words, "why wouldn't I merge the indicies into one big index every time?" or "why wouldn't I make every index a delta index?"

I'm still not sure it is clear from the comment.