C APIのE2Eテストを作る #425

qryxip · 2023-02-05T17:20:24Z

内容

/crates/voicevox_core/src/test_util.rsを/crates/test_utilに分離する
~~/crates/test_util/src/bin/run-cdylib.rsを書き、~~ assert_cmdでDLLのE2Eテストをする機構を用意する
DLLのE2Eテストを書く

その他

qryxip · 2023-02-05T18:00:05Z

よく考えたらこれだとE2Eテストの内容がtest_utils側に来てしまいますね。
bin用クレートは用意しないでrusty-fork風にどうにかしてtestバイナリ自体を利用できないか考えた方がいいかも。

qryxip · 2023-02-06T01:42:07Z

あ、わかった単純にharness外してバイナリを再帰的に呼べばいいですね。多分。

qryxip · 2023-02-06T02:34:57Z

いやharnessが無くなるとそれはそれで面倒が生じる、と思ったのですが、代替harnessとしてlibtest-mimicというお誂え向きなものがありますね。これ使うのがいいかも。

参考: Custom test harnesses - cargo-nextest

qryxip · 2023-02-06T14:27:24Z

テスト機構はとりあえずできました。

const char* voicevox_get_version(void)に対するテストが以下のように書けます。

use std::ffi::CStr;

use assert_cmd::assert::Assert;

use crate::Symbols;

pub(crate) unsafe fn exec(
    Symbols {
        voicevox_get_version,
    }: Symbols<'_>,
) -> anyhow::Result<()> {
    let version = voicevox_get_version();
    let version = CStr::from_ptr(version).to_str()?;
    println!("Version: {version:?}");
    Ok(())
}

pub(crate) fn assert(assert: Assert) {
    assert.success().stdout("Version: \"0.0.0\"\n").stderr("");
}

(追記) よく考えたらexec側でassert!すればいいですね。

テストが成功した場合:

テストが失敗した場合:

qryxip · 2023-02-08T14:41:51Z

snapshots.tomlというファイルにAudioQueryやwavのSHA-256を書き、それをチェックするという方針にしました。
~~OSS版モデルと0.14.1のモデルの両方をチェックしてしまうことを考えています。~~
暗号化があるから製品版モデルだけじゃ駄目でしたね…DLLごと持ってきたとしても何に対するテストなのかという話になってしまう。

[metas]
metas-json = "675be97048d261b06ae5fe75a2614ba253422abdcc9eb90ad4a073875af0f57e"

use std::ffi::CStr;

use assert_cmd::assert::{Assert, AssertResult};
use serde::Deserialize;

use crate::Symbols;

use super::SNAPSHOTS;

#[derive(Deserialize)]
#[serde(rename_all = "kebab-case")]
pub(super) struct Snapshots {
    metas_json: String,
}

pub(crate) unsafe fn exec(Symbols { metas, .. }: Symbols<'_>) -> anyhow::Result<()> {
    let metas_json = metas();
    let metas_json = CStr::from_ptr(metas_json).to_str()?;
    std::assert_eq!(SNAPSHOTS.metas.metas_json, super::sha256(metas_json));
    Ok(())
}

pub(crate) fn assert_output(assert: Assert) -> AssertResult {
    assert.try_success()?.try_stdout("")?.try_stderr("")
}

qryxip · 2023-02-08T15:06:41Z

あーautocrlf!!!

https://github.com/VOICEVOX/voicevox_core/actions/runs/4125239867

qryxip · 2023-02-10T16:11:40Z

tracing-subscriberのタイムスタンプのマスクもできたし、あとはyukarin_sa_forwardとdecode_forwardをやれば形にはなりそう。

qryxip · 2023-02-10T16:23:09Z

"It is not used by any node and should be removed from the model"がActions環境だけ出てない...???

https://github.com/VOICEVOX/voicevox_core/actions/runs/4145617823/jobs/7170233703

This reverts commit 6473c8d.

This reverts commit dbc99c9.

qryxip · 2023-02-10T19:22:01Z

あ! #423 があるから"It is not used by any node and should be removed from the model"が無いのが正しいんでした。
git pullを忘れて入ってませんでした。

qryxip · 2023-03-09T17:20:20Z

そういえば今のRustならこんなこともできますね。その発想はありませんでした。

mod initialize前にモデルを読み込むとエラーになるテスト;
mod エンジンを起動してyukarin_sとyukarin_saとdecodeの推論を行う; // '・'や，や'、'は利用不可

// これもOK
#[path = "./testcases/エンジンを起動してyukarin_s・yukarin_sa・decodeの推論を行う.rs"]
mod _0; 
#[path = "./testcases/initialize前にモデルを読み込むとエラーになるテスト.rs"]
mod _1;

// あるいはこれでもOK

const _: () = {
    #[path = "./testcases/エンジンを起動してyukarin_s・yukarin_sa・decodeの推論を行う.rs"]
    mod _testcase;
};

const _: () = {
    #[path = "./testcases/initialize前にモデルを読み込むとエラーになるテスト.rs"]
    mod _testcase;
};

(追記) いやrust-analyzerが混乱しそうではありますが...

qryxip · 2023-03-09T17:49:58Z

あと、他の人が迷子にならないように、assert_cdylib.exec、snapshots.section、Symbols辺りの一般的な単語を使っているクラス名のものは１行くらいで良いので説明コメントがあれば、それだけでもかなり読みやすくなる気がしました！！

入れてみました。

Hiroshiba

LGTM！！！！！！

ありがとうございました！！

コメントにsuggestコメントしてみました！！（認識合わせも兼ねて）

crates/voicevox_core_c_api/tests/e2e/assert_cdylib.rs

crates/voicevox_core_c_api/tests/e2e/snapshots.rs

Hiroshiba · 2023-03-10T18:40:07Z

あと１名Rustに明るい方にレビューをお願いできるととても心強いです･･･！！

@PickledChair さん、 @qwerty2501 さんお願いできますか 🙇
1000行ほどあるのでちょっとだけ共有をば。

e2eテストの入り口はe2e/main.rs
e2e/testcasesに各テストケースを書く
stdoutログなどのテストも含むため各テストは別プロセスで起動する
各テストが実行されるか・ログが意図通りかがテストされる

Co-authored-by: Hiroshiba <[email protected]>

crates/voicevox_core_c_api/tests/e2e/snapshots.rs

qryxip · 2023-03-14T16:13:06Z

#370 見てたらサニタイザを挟みたくなってきました…
ONNX RuntimeやOpen JTalkとかのバインディング及び本体自体の課題も検出できるはず。
LLVMのやつはNightly Rustを使う必要があるので面倒ですが、Valgrindならバイナリひとつ持ってきて挟むだけでいいはず。

qryxip · 2023-03-18T03:21:40Z

LLVMのやつ(-Z sanitizer)の方がいい気がしてきました。単体テストも対象にできますし。
Nightly Rustを管理する必要がありますが、Nightlyなら-Z avoid-dev-depsとかcargo-udepsとか使えるし悪くないかと。
その場合このPRじゃなくて別PRでの導入ですね。

qryxip · 2023-03-24T02:06:00Z

VOICEVOX/onnxruntime-rs#16によって

こんな感じ

{timestamp}  INFO onnxruntime::onnxruntime: "Creating and using per session threadpools since use_per_session_threads_ is true"
{timestamp}  INFO onnxruntime::onnxruntime: "Dynamic block base set to 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Initializing session."
{timestamp}  INFO onnxruntime::onnxruntime: "Adding default CPU execution provider."
{timestamp}  INFO onnxruntime::onnxruntime: "Creating BFCArena for Cpu with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 memory limit: 18446744073709551615 arena_extend_strategy: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Total fused reshape node count: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Saving initialized tensors."
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:0 (requested) num_bytes: 24 (actual) rounded_bytes:256"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 1048576 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 1048576"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd97feff040 to 0x7fd97ffff040"
{timestamp}  INFO onnxruntime::onnxruntime: "Done saving initialized tensors"
{timestamp}  INFO onnxruntime::onnxruntime: "Session successfully initialized."
{timestamp}  INFO onnxruntime::onnxruntime: "Creating and using per session threadpools since use_per_session_threads_ is true"
{timestamp}  INFO onnxruntime::onnxruntime: "Dynamic block base set to 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Initializing session."
{timestamp}  INFO onnxruntime::onnxruntime: "Adding default CPU execution provider."
{timestamp}  INFO onnxruntime::onnxruntime: "Creating BFCArena for Cpu with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 memory limit: 18446744073709551615 arena_extend_strategy: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Total fused reshape node count: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Total fused reshape node count: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Saving initialized tensors."
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:0 (requested) num_bytes: 8 (actual) rounded_bytes:256"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 1048576 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 1048576"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd97d5f9040 to 0x7fd97d6f9040"
{timestamp}  INFO onnxruntime::onnxruntime: "Done saving initialized tensors"
{timestamp}  INFO onnxruntime::onnxruntime: "Saving initialized tensors."
{timestamp}  INFO onnxruntime::onnxruntime: "Done saving initialized tensors"
{timestamp}  INFO onnxruntime::onnxruntime: "Subgraph input with name cond is not used by any node."
{timestamp}  INFO onnxruntime::onnxruntime: "Session successfully initialized."
{timestamp}  INFO onnxruntime::onnxruntime: "Creating and using per session threadpools since use_per_session_threads_ is true"
{timestamp}  INFO onnxruntime::onnxruntime: "Dynamic block base set to 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Initializing session."
{timestamp}  INFO onnxruntime::onnxruntime: "Adding default CPU execution provider."
{timestamp}  INFO onnxruntime::onnxruntime: "Creating BFCArena for Cpu with following configs: initial_chunk_size_bytes: 1048576 max_dead_bytes_per_chunk: 134217728 initial_growth_chunk_size_bytes: 2097152 memory limit: 18446744073709551615 arena_extend_strategy: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Total fused reshape node count: 0"
{timestamp}  INFO onnxruntime::onnxruntime: "Saving initialized tensors."
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:0 (requested) num_bytes: 8 (actual) rounded_bytes:256"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 1048576 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 1048576"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd955d7d040 to 0x7fd955e7d040"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:0 (requested) num_bytes: 192 (actual) rounded_bytes:256"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 2097152 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 3145728"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd955b7c040 to 0x7fd955d7c040"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:13 (requested) num_bytes: 2883584 (actual) rounded_bytes:2883584"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 4194304 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 7340032"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd95577b040 to 0x7fd955b7b040"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:0 (requested) num_bytes: 24 (actual) rounded_bytes:256"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 8388608 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 15728640"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd954f7a040 to 0x7fd95577a040"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:15 (requested) num_bytes: 8388608 (actual) rounded_bytes:8388608"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 16777216 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 32505856"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd953f79040 to 0x7fd954f79040"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:13 (requested) num_bytes: 2883584 (actual) rounded_bytes:2883584"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 33554432 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 66060288"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd951f78040 to 0x7fd953f78040"
{timestamp}  INFO onnxruntime::onnxruntime: "Done saving initialized tensors"
{timestamp}  INFO onnxruntime::onnxruntime: "Extending BFCArena for Cpu. bin_num:15 (requested) num_bytes: 8388608 (actual) rounded_bytes:8388608"
{timestamp}  INFO onnxruntime::onnxruntime: "Extended allocation by 67108864 bytes."
{timestamp}  INFO onnxruntime::onnxruntime: "Total allocated bytes: 133169152"
{timestamp}  INFO onnxruntime::onnxruntime: "Allocated memory at 0x7fd94df77040 to 0x7fd951f77040"
{timestamp}  INFO onnxruntime::onnxruntime: "Session successfully initialized."
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"
{timestamp}  INFO onnxruntime::onnxruntime: "Begin execution"

のメッセージが出るようになったのですが、releaseプロファイルでのビルドなら出なくなるため--releaseのDLLをテストしてしまうことにしました。
"E2E"なのだからこっちの方が実態に即しているはず。

PickledChair

すみません、かなりお待たせしてしまいましたが、LGTM です！（自分の知らない有用な crate やアイディアが豊富だったので、全体を理解するのに時間がかかってしまいました……）

テストの追加もやりやすそうになっており、ありがたいです。

Hiroshiba · 2023-04-08T18:34:30Z

マージします！！

test_utilをクレートに分離する

54780bd

テスト機構を作る

f687c39

qryxip added 5 commits February 6, 2023 23:51

Actions上で動かす

700f406

chore: assert → assert_output

4f5d90f

assert_cmdのエラーはlibtest_mimic::Failed経由で表示する

65dfc39

得られた値はassert_eq!でチェックする

9905239

metas()をテスト

b4a8f1a

yukarin_s_forwardまで

5804f0d

qryxip added 8 commits February 11, 2023 02:47

相対パスで十分

3fb12d7

cargo build -p voicevox_core_c_api --libを単体で実行

6473c8d

Revert "cargo build -p voicevox_core_c_api --libを単体で実行"

454a194

This reverts commit 6473c8d.

onnxruntime/disable-sys-build-scriptをオンにする

dbc99c9

Revert "onnxruntime/disable-sys-build-scriptをオンにする"

cb8d777

This reverts commit dbc99c9.

ORT_OUT_DIR=にする

a492f54

不要な-pを削除

f2420af

Merge branch 'main' into e2e-tests-for-c-api

1ed805a

qryxip added 5 commits February 11, 2023 04:23

VOICEVOX#423 に対応

a9b0c1b

list_windows_video_cardsの結果をマスク

e639054

Minor refactor

a65e435

cargo buildのスキップについてコメント

15384ac

voicevox_get_versionを削除

960cdc4

mask → log_mask

3698e3a

docstringを追加

384dd52

Hiroshiba approved these changes Mar 10, 2023

View reviewed changes

crates/voicevox_core_c_api/tests/e2e/assert_cdylib.rs Outdated Show resolved Hide resolved

crates/voicevox_core_c_api/tests/e2e/assert_cdylib.rs Outdated Show resolved Hide resolved

crates/voicevox_core_c_api/tests/e2e/snapshots.rs Outdated Show resolved Hide resolved

qryxip and others added 5 commits March 11, 2023 03:47

コメントを更新

303d6fc

Co-authored-by: Hiroshiba <[email protected]>

コメントを更新

3f4ecd4

Co-authored-by: Hiroshiba <[email protected]>

コメントを更新

1a97661

Co-authored-by: Hiroshiba <[email protected]>

trailing commaが残っていたのを削除

04d575c

case!にdocを書く

9a4525b

qryxip commented Mar 11, 2023

View reviewed changes

crates/voicevox_core_c_api/tests/e2e/snapshots.rs Show resolved Hide resolved

qryxip requested review from qwerty2501 and PickledChair March 18, 2023 03:21

qryxip added 3 commits March 20, 2023 18:42

Merge branch 'main' into e2e-tests-for-c-api

a45dcf6

Merge branch 'main' into e2e-tests-for-c-api

a299929

releaseのDLLをテストする

0d9a0b4

Merge branch 'main' into e2e-tests-for-c-api

fa76786

PickledChair approved these changes Apr 8, 2023

View reviewed changes

Hiroshiba merged commit b6e7e86 into VOICEVOX:main Apr 8, 2023

qryxip mentioned this pull request Apr 19, 2023

group_imports=StdExternalCrateでRustfmtをかける #466

Closed

3 tasks

qryxip mentioned this pull request Apr 27, 2023

新クラス設計API #370

Merged

sevenc-nanashi mentioned this pull request May 6, 2023

Python APIのテストを作る #482

Closed

qryxip mentioned this pull request May 27, 2023

[project-vvm-async-api] いくつかのC関数を定数にする #503

Merged

2 tasks

qryxip mentioned this pull request Jun 4, 2023

Rust APIのbuild.rsを抹消する #508

Merged

qryxip mentioned this pull request Jan 2, 2024

モーフィング機能を追加する #713

Open

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

C APIのE2Eテストを作る #425

C APIのE2Eテストを作る #425

qryxip commented Feb 5, 2023 •

edited

Loading

qryxip commented Feb 5, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 8, 2023 •

edited

Loading

qryxip commented Feb 8, 2023

qryxip commented Feb 10, 2023

qryxip commented Feb 10, 2023

qryxip commented Feb 10, 2023

qryxip commented Mar 9, 2023 •

edited

Loading

qryxip commented Mar 9, 2023

Hiroshiba left a comment

Hiroshiba commented Mar 10, 2023

qryxip commented Mar 14, 2023 •

edited

Loading

qryxip commented Mar 18, 2023 •

edited

Loading

qryxip commented Mar 24, 2023 •

edited

Loading

PickledChair left a comment

Hiroshiba commented Apr 8, 2023

C APIのE2Eテストを作る #425

C APIのE2Eテストを作る #425

Conversation

qryxip commented Feb 5, 2023 • edited Loading

内容

関連 Issue

その他

qryxip commented Feb 5, 2023 • edited Loading

qryxip commented Feb 6, 2023 • edited Loading

qryxip commented Feb 6, 2023 • edited Loading

qryxip commented Feb 6, 2023 • edited Loading

qryxip commented Feb 8, 2023 • edited Loading

qryxip commented Feb 8, 2023

qryxip commented Feb 10, 2023

qryxip commented Feb 10, 2023

qryxip commented Feb 10, 2023

qryxip commented Mar 9, 2023 • edited Loading

qryxip commented Mar 9, 2023

Hiroshiba left a comment

Choose a reason for hiding this comment

Hiroshiba commented Mar 10, 2023

qryxip commented Mar 14, 2023 • edited Loading

qryxip commented Mar 18, 2023 • edited Loading

qryxip commented Mar 24, 2023 • edited Loading

PickledChair left a comment

Choose a reason for hiding this comment

Hiroshiba commented Apr 8, 2023

qryxip commented Feb 5, 2023 •

edited

Loading

qryxip commented Feb 5, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 6, 2023 •

edited

Loading

qryxip commented Feb 8, 2023 •

edited

Loading

qryxip commented Mar 9, 2023 •

edited

Loading

qryxip commented Mar 14, 2023 •

edited

Loading

qryxip commented Mar 18, 2023 •

edited

Loading

qryxip commented Mar 24, 2023 •

edited

Loading