Skip to content

Build projects in wamr-test-suites with multiple cores #811

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 19 commits into from
Oct 30, 2021
Merged

Build projects in wamr-test-suites with multiple cores #811

Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
05e315e
Merge pull request #360 from bytecodealliance/main
wenyongh Oct 13, 2021
d4f2649
Apply clang-format for core/shared and product-mini files
wenyongh Oct 13, 2021
8c3ccef
Merge pull request #361 from bytecodealliance/main
wenyongh Oct 14, 2021
084fef1
Merge pull request #362 from bytecodealliance/main
wenyongh Oct 14, 2021
49fbb9c
Merge pull request #363 from bytecodealliance/main
wenyongh Oct 15, 2021
f0cd7ea
Fix app heap corrupted unchecked issue
wenyongh Oct 15, 2021
a4c8883
Change API to wasm_runtime_show_app_heap_corrupted_prompt()
wenyongh Oct 15, 2021
0f25f37
Merge pull request #364 from bytecodealliance/main
wenyongh Oct 15, 2021
cb984fe
Merge pull request #365 from bytecodealliance/main
wenyongh Oct 19, 2021
5a39a9d
Apply clang-format for more files
wenyongh Oct 21, 2021
d7857c5
Merge pull request #366 from bytecodealliance/main
wenyongh Oct 21, 2021
beaaf52
Enable auto install wasi-sdk when build wamr-sdk
wenyongh Oct 22, 2021
d2867ca
Merge bytecodealliance:main into wenyongh:main
wenyongh Oct 26, 2021
3a68634
Merge pull request #367 from bytecodealliance/main
wenyongh Oct 26, 2021
4c63230
Merge pull request #368 from bytecodealliance/main
wenyongh Oct 27, 2021
6e1f4c9
Fix workflow issue, fix dockerfile bug, refine code format
wenyongh Oct 30, 2021
d44e6b0
Refine code format
wenyongh Oct 30, 2021
3fca5ff
Merge pull request #369 from bytecodealliance/main
wenyongh Oct 30, 2021
13bdd8c
Build projets in wamr-test-suites with multiple cores
wenyongh Oct 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions tests/wamr-test-suites/test_wamr.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ function unit_test()

# keep going and do not care if it is success or not
make -ki clean | true
cmake ${compile_flag} ${WORK_DIR}/../../unit && make
cmake ${compile_flag} ${WORK_DIR}/../../unit && make -j 4
Copy link
Collaborator

@xujuntwt95329 xujuntwt95329 Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

had better use $(nproc)

Suggested change
cmake ${compile_flag} ${WORK_DIR}/../../unit && make -j 4
cmake ${compile_flag} ${WORK_DIR}/../../unit && make -j $(nproc)

Copy link
Collaborator

@lum1n0us lum1n0us Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nproc may not be installed. 4 might be a safe option

Copy link
Collaborator

@xujuntwt95329 xujuntwt95329 Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, thanks.

Copy link
Contributor Author

@wenyongh wenyongh Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, so let's use -j 4.
Another place to reduce the workloads compilation time is building iwasm in compilation_on_android_ubuntu_macos.yml, currently we get llvm libraries
before building iwasm for each mode, in fact we only need to get llvm libraries
for the mode jit and lazyjit. For fast/classic interpreter mode and aot mode,
we don't need to get the llvm libraires, is it easy to add condition check
in the scripts?

- name: Get LLVM libraries
        id: cache_llvm
        if: ${{ matrix.light == 'green' }}
        uses: actions/cache@v2
        with:
          path: |
            ./core/deps/llvm/build/bin
            ./core/deps/llvm/build/include
            ./core/deps/llvm/build/lib
            ./core/deps/llvm/build/libexec
            ./core/deps/llvm/build/share
          key: ${{ matrix.os }}-${{ env.LLVM_CACHE_SUFFIX }}

e.g. modify this line: if: ${{ matrix.light == 'green' }} to something like:

if: ${{ matrix.light == 'green'
        && (make_options_run_mode == $JIT_BUILD_OPTIONS
            || make_options_run_mode == $LAZY_JIT_BUILD_OPTIONS)
     }}

Copy link
Collaborator

@lum1n0us lum1n0us Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Time consumption is defined by the longest job in parallel mode. In our workflow, the followings will not start until all five build_iwasm jobs are finished.

Copy link
Contributor Author

@wenyongh wenyongh Oct 31, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There should be total 243 build_iwasm jobs but not five in compilation_on_android_ubuntu_macos.yml:
https://github.com/bytecodealliance/wasm-micro-runtime/actions/runs/1402357515
build_iwasm
It is the longest phase in all the workflows.
Currently each build_iwasm job will get LLVM libraries firstly which costs some time, but there
are a lot of interpreter/aot mode jobs which don't need llvm libraries, if we can skip this step for them,
we should be able to reduce the total time of 243 build_iwasm jobs, so as to reduce the total time of
all workflows.

if [ "$?" != 0 ];then
echo -e "\033[31mbuild unit test failed, you may need to change wamr into dev/aot branch and ensure llvm is built \033[0m"
exit 1
Expand Down Expand Up @@ -311,7 +311,7 @@ function spec_test()

echo "compile the reference intepreter"
pushd interpreter
make opt
make opt -j 4
popd

git apply ../../spec-test-script/simd_ignore_cases.patch
Expand Down Expand Up @@ -359,7 +359,7 @@ function spec_test()
git pull
git reset --hard origin/main
cd ..
make -C wabt gcc-release
make -C wabt gcc-release -j 4
fi

ln -sf ${WORK_DIR}/../spec-test-script/all.sh .
Expand Down Expand Up @@ -473,7 +473,7 @@ function build_iwasm_with_cfg()
&& if [ -d build ]; then rm -rf build/*; else mkdir build; fi \
&& cd build \
&& cmake $* .. \
&& make
&& make -j 4
cd ${WAMR_DIR}/product-mini/platforms/linux-sgx/enclave-sample \
&& make clean \
&& make SPEC_TEST=1
Expand All @@ -482,7 +482,7 @@ function build_iwasm_with_cfg()
&& if [ -d build ]; then rm -rf build/*; else mkdir build; fi \
&& cd build \
&& cmake $* .. \
&& make
&& make -j 4
fi

if [ "$?" != 0 ];then
Expand All @@ -506,7 +506,7 @@ function build_wamrc()
&& if [ -d build ]; then rm -r build/*; else mkdir build; fi \
&& cd build \
&& cmake .. \
&& make
&& make -j 4
Copy link
Collaborator

@xujuntwt95329 xujuntwt95329 Oct 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above

}

### Need to add a test suite?
Expand Down