panics: index out of bounds when using tokio runtime

[joelchen]

When using RandomForestClassifier/RandomForestRegressor with either DenseMatrix or ndarray, smartcore panics with out of bounds, as it always access Vec at index number of rows. Happened with my initial X dataset of 75600000 rows 72 columns and Y dataset of 75600000 rows 1 column, as well as a smaller X dataset of 12441600 rows 72 columns and Y dataset of 12441600 rows 1 column. Following are the full backtraces.

RandomForestClassifier:

length: 895795200, rows: 12441600

thread 'main' panicked at /home/joel/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/smartcore-0.4.1/src/linalg/basic/vector.rs:39:14:
index out of bounds: the len is 12441600 but the index is 12441600
stack backtrace:
   0:     0x7f6988282b42 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h055aa7e8f655acd3
   1:     0x7f69882bbaa3 - core::fmt::write::hc7be0cf463b7740b
   2:     0x7f69882803e3 - std::io::Write::write_fmt::he7748fe82a9ccc5c
   3:     0x7f6988282992 - std::sys::backtrace::BacktraceLock::print::h606bd811d3a62961
   4:     0x7f6988283af2 - std::panicking::default_hook::{{closure}}::h888745fb686c27f7
   5:     0x7f69882838f5 - std::panicking::default_hook::ha1b658d34a68f564
   6:     0x7f6988284492 - std::panicking::rust_panic_with_hook::h5781b1ec8ca7c630
   7:     0x7f698828423a - std::panicking::begin_panic_handler::{{closure}}::hac038ec9b0ef9af3
   8:     0x7f6988283049 - std::sys::backtrace::__rust_end_short_backtrace::hcb23fc0c659afbb2
   9:     0x7f6988283ecd - __rustc[95feac21a9532783]::rust_begin_unwind
  10:     0x7f6988211c90 - core::panicking::panic_fmt::hc4d2bb11085703b5
  11:     0x7f6988211e72 - core::panicking::panic_bounds_check::h771968d8b46b7b5d
  12:     0x7f6988213810 - smartcore::linalg::basic::arrays::MutArrayView1::argsort_mut::h53d95e0cbb05408a
  13:     0x7f6988218861 - smartcore::tree::decision_tree_classifier::DecisionTreeClassifier<TX,TY,X,Y>::fit_weak_learner::h95da140acfa4e5d5
  14:     0x7f698821dfed - smartcore::ensemble::random_forest_classifier::RandomForestClassifier<TX,TY,X,Y>::fit::h6e0704da24efcfe7
  15:     0x7f6988228062 - tokio::runtime::park::CachedParkThread::block_on::h64b20800be8018e4
  16:     0x7f698822b15c - tokio::runtime::context::runtime::enter_runtime::h4d2ba6f21555d3e9
  17:     0x7f698821ea5f - tokio::runtime::runtime::Runtime::block_on::h4a6db00acf1ca95b
  18:     0x7f698821c448 - application::main::h468474553be97be8
  19:     0x7f6988226e63 - std::sys::backtrace::__rust_begin_short_backtrace::h01d09b3113aba70a
  20:     0x7f6988225959 - std::rt::lang_start::{{closure}}::hf1a4e4749d57fbc4
  21:     0x7f698827a3a0 - std::rt::lang_start_internal::hd8083bca76e1233a
  22:     0x7f698821c505 - main

RandomForestRegressor:

length: 895795200, rows: 12441600

thread 'main' panicked at /home/joel/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/smartcore-0.4.1/src/linalg/basic/vector.rs:39:14:
index out of bounds: the len is 12441600 but the index is 12441600
stack backtrace:
   0:     0x7f39333e5702 - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h055aa7e8f655acd3
   1:     0x7f393341e353 - core::fmt::write::hc7be0cf463b7740b
   2:     0x7f39333e2fa3 - std::io::Write::write_fmt::he7748fe82a9ccc5c
   3:     0x7f39333e5552 - std::sys::backtrace::BacktraceLock::print::h606bd811d3a62961
   4:     0x7f39333e66b2 - std::panicking::default_hook::{{closure}}::h888745fb686c27f7
   5:     0x7f39333e64b5 - std::panicking::default_hook::ha1b658d34a68f564
   6:     0x7f39333e7052 - std::panicking::rust_panic_with_hook::h5781b1ec8ca7c630
   7:     0x7f39333e6dfa - std::panicking::begin_panic_handler::{{closure}}::hac038ec9b0ef9af3
   8:     0x7f39333e5c09 - std::sys::backtrace::__rust_end_short_backtrace::hcb23fc0c659afbb2
   9:     0x7f39333e6a8d - __rustc[95feac21a9532783]::rust_begin_unwind
  10:     0x7f3933378700 - core::panicking::panic_fmt::hc4d2bb11085703b5
  11:     0x7f39333788e2 - core::panicking::panic_bounds_check::h771968d8b46b7b5d
  12:     0x7f393337ce10 - smartcore::linalg::basic::arrays::MutArrayView1::argsort_mut::h53d95e0cbb05408a
  13:     0x7f393337e211 - smartcore::tree::decision_tree_regressor::DecisionTreeRegressor<TX,TY,X,Y>::fit_weak_learner::h9d44361e42916a87
  14:     0x7f393338d6e3 - smartcore::ensemble::random_forest_regressor::RandomForestRegressor<TX,TY,X,Y>::fit::h2c4fedec4040839d
  15:     0x7f3933387582 - tokio::runtime::park::CachedParkThread::block_on::h64b20800be8018e4
  16:     0x7f393338ee2c - tokio::runtime::runtime::Runtime::block_on::h4a6db00acf1ca95b
  17:     0x7f393338bee8 - application::main::h468474553be97be8
  18:     0x7f39333897c3 - std::sys::backtrace::__rust_begin_short_backtrace::h01d09b3113aba70a
  19:     0x7f3933386299 - std::rt::lang_start::{{closure}}::hf1a4e4749d57fbc4
  20:     0x7f39333dcf60 - std::rt::lang_start_internal::hd8083bca76e1233a
  21:     0x7f393338bfa5 - main

[Mec-iS]

Hello, thanks for this report.

I am trying to understand if this is a regression. Can you try your code with version 0.3 as well and let me know?

EDIT:
or also provide a test case that fails would be great.

I have tried to run a test for large arrays in a single machine and all the methods listed in your stack trace work.

Also please consider: the async runtime isn't directly responsible for the panic, but its task scheduling might expose edge cases in smartCore's code or in memory allocation. Consider testing with:

#[tokio::main(flavor = "current_thread")]

and let me know if this also panics.

I tried:

    #[test]
    fn fit_large() {
        let x = DenseMatrix::<f64>::rand(50000, 50000);
        let y: Vec<f64> = vec![
            0.5; 50000 as usize
        ];

        let y_hat = DecisionTreeRegressor::fit(&x, &y, Default::default())
            .and_then(|t| t.predict(&x))
            .unwrap();

        for i in 0..y_hat.len() {
            assert!((y_hat[i] - y[i]).abs() < 0.1);
        }

        DecisionTreeRegressor::fit(
            &x,
            &y,
            DecisionTreeRegressorParameters {
                max_depth: Option::None,
                min_samples_leaf: 2,
                min_samples_split: 6,
                seed: Option::None,
            },
        )
        .and_then(|t| t.predict(&x))
        .unwrap();

        assert!(true);
    }

Also with these other constants is passing:

...
        let x = DenseMatrix::<f64>::rand(15000000, 2);
        let y: Vec<f64> = vec![
            0.5; 15000000 as usize
        ];
...

UPDATE:

smartcore is meant to be run in single-threaded fashion, this code works with large matrices:

use smartcore::linalg::basic::matrix::DenseMatrix;
use smartcore::linalg::basic::arrays::{Array1, Array2, MutArrayView1};
use smartcore::tree::decision_tree_regressor::{DecisionTreeRegressor, DecisionTreeRegressorParameters};

#[tokio::main]
async fn main() {
    // Run the CPU-bound fit_large test in a blocking thread
    tokio::task::spawn_blocking(|| {
        fit_large();
    })
    .await
    .unwrap();
}

fn fit_large() {
    let x = DenseMatrix::<f64>::rand(15_000_000, 2);
    let y: Vec<f64> = vec![0.5; 15_000_000];

    let y_hat = DecisionTreeRegressor::fit(&x, &y, Default::default())
        .and_then(|t| t.predict(&x))
        .unwrap();

    for i in 0..y_hat.len() {
        assert!((y_hat[i] - y[i]).abs() < 0.1);
    }

    DecisionTreeRegressor::fit(
        &x,
        &y,
        DecisionTreeRegressorParameters {
            max_depth: None,
            min_samples_leaf: 2,
            min_samples_split: 6,
            seed: None,
        },
    )
    .and_then(|t| t.predict(&x))
    .unwrap();

    println!("{:?}", true);
}

What would it take to make smartcore working with async? it would take dedicated methods and a total rewrite of smartcore to make it works in distributed fashion. If you want to do it we may be interested in considering a smartcore-tokio package.

Conclusion

• smartCore is not async and cannot be made async without a major rewrite.
• For now, always use tokio::task::spawn_blocking for smartCore operations in async code.
• Full async/await support in ML libraries is rare and generally not needed for CPU-bound tasks, except for orchestration and I/O.

@morenol any hint on this?

[Mec-iS]

Debugging this helped me finding this missing type declaration:

611a831

may or may improve things for you. Will be part of the next release.
It would be helpful if you can try installing from the branch and try it on your code.