Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[MXNET-651] MXNet Model Backwards Compatibility Checker #11626

Merged
merged 61 commits into from
Jul 31, 2018
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
4ee8b21
Added MNIST-MLP-Module-API models to check model save and load_checkp…
piyushghai Jul 6, 2018
118850f
Added LENET with Conv2D operator training file
piyushghai Jul 6, 2018
27863fd
Added LENET with Conv2d operator inference file
piyushghai Jul 6, 2018
b3e9774
Added LanguageModelling with RNN training file
piyushghai Jul 7, 2018
c141701
Added LamguageModelling with RNN inference file
piyushghai Jul 7, 2018
35cbefb
Added hybridized LENET Gluon Model training file
piyushghai Jul 9, 2018
418f805
Added hybridized LENET gluon model inference file
piyushghai Jul 9, 2018
600efaf
Added license headers
piyushghai Jul 9, 2018
d73b9e2
Refactored the model and inference files and extracted out duplicate …
piyushghai Jul 9, 2018
3eeba08
Added runtime function for executing the MBCC files
piyushghai Jul 10, 2018
9c0157c
Added JenkinsFile for MBCC to be run as a nightly job
piyushghai Jul 10, 2018
3d43bcd
Added boto3 install for s3 uploads
piyushghai Jul 10, 2018
4b70e4a
Added README for MBCC
piyushghai Jul 10, 2018
08ad342
Added license header
piyushghai Jul 10, 2018
5d1c3fc
Added more common functions from lm_rnn_gluon_train and inference fil…
piyushghai Jul 10, 2018
cfe8dfc
Added scripts for training models on older versions of MXNet
piyushghai Jul 11, 2018
7c41488
Added check for preventing inference script from crashing in case no …
piyushghai Jul 11, 2018
50be5d8
Fixed indentation issue
piyushghai Jul 11, 2018
c3c9129
Replaced Penn Tree Bank Dataset with Sherlock Holmes Dataset
piyushghai Jul 11, 2018
3485352
Fixed indentation issue
piyushghai Jul 11, 2018
af9b86d
Removed training in models and added smaller models. Now we are simpl…
piyushghai Jul 12, 2018
79cfa46
Updated README
piyushghai Jul 12, 2018
4df779b
Fixed indentation error
piyushghai Jul 12, 2018
04465b0
Fixed indentation error
piyushghai Jul 12, 2018
2d5cf09
Removed code duplication in the training file
piyushghai Jul 13, 2018
7bfdf87
Added comments for runtime_functions script for training files
piyushghai Jul 16, 2018
c80ee31
Merged S3 Buckets for storing data and models into one
piyushghai Jul 16, 2018
e764d5a
Automated the process to fetch MXNet versions from git tags
piyushghai Jul 16, 2018
05ded05
Added defensive checks for the case where the data might not be found
piyushghai Jul 16, 2018
60c7be0
Fixed issue where we were performing inference on state model files
piyushghai Jul 16, 2018
9d4d099
Replaced print statements with logging ones
piyushghai Jul 18, 2018
d08ba5a
Merge branch 'master' into mbcc
piyushghai Jul 25, 2018
cebfb26
Removed boto install statements and move them into ubuntu_python docker
piyushghai Jul 25, 2018
f7a36eb
Separated training and uploading of models into separate files so tha…
piyushghai Jul 25, 2018
1f63941
Updated comments and README
piyushghai Jul 26, 2018
fbaf3e0
Fixed pylint warnings
piyushghai Jul 26, 2018
edd6816
Removed the venv for training process
piyushghai Jul 26, 2018
87103d4
Fixed indentation in the MBCC Jenkins file and also separated out tra…
piyushghai Jul 26, 2018
eb24e8e
Fixed indendation
piyushghai Jul 26, 2018
3525656
Fixed erroneous single quote
piyushghai Jul 26, 2018
25e7ec7
Added --user flag to check for Jenkins error
piyushghai Jul 26, 2018
00ee6e7
Removed unused methods
piyushghai Jul 26, 2018
a3a72b8
Added force flag in the pip command to install mxnet
piyushghai Jul 26, 2018
86e8882
Removed the force-re-install flag
piyushghai Jul 26, 2018
ddb672a
Changed exit 1 to exit 0
piyushghai Jul 26, 2018
9e77064
Added quotes around the shell command
piyushghai Jul 26, 2018
69843fb
added packlibs and unpack libs for MXNet builds
piyushghai Jul 26, 2018
fae44fe
Changed PythonPath from relative to absolute
piyushghai Jul 27, 2018
c099979
Created dedicated bucket with correct permission
marcoabreu Jul 30, 2018
ffcc637
Fix for python path in training
piyushghai Jul 30, 2018
7f7f6e3
Merge branch 'mbcc' of https://github.com/piyushghai/incubator-mxnet …
piyushghai Jul 30, 2018
33096c0
Changed bucket name to CI bucket
piyushghai Jul 30, 2018
8a085b5
Added set -ex to the upload shell script
piyushghai Jul 30, 2018
5207ab1
Now raising an exception if no models are found in the S3 bucket
piyushghai Jul 30, 2018
5e30f7a
Added regex to train models script
piyushghai Jul 30, 2018
e079d3c
Added check for performing inference only on models trained on same m…
piyushghai Jul 30, 2018
ceac705
Added set -ex flags to shell scripts
piyushghai Jul 30, 2018
16d320a
Added multi-version regex checks in training
piyushghai Jul 30, 2018
19495d6
Fixed typo in regex
piyushghai Jul 30, 2018
d8fa75d
Now we will train models for all the minor versions for a given major…
piyushghai Jul 30, 2018
ca01aa2
Added check for validating current_version
piyushghai Jul 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ def docker_run(platform, function_name, use_nvidia, shared_mem = '500m') {

try {
stage('MBCC Train'){
node('mxnetlinux-cpu') {
node('restricted-mxnetlinux-cpu') {
ws('workspace/modelBackwardsCompat') {
init_git()
// Train models on older versions
Expand All @@ -81,7 +81,7 @@ try {
}

stage('MXNet Build'){
node('mxnetlinux-cpu') {
node('restricted-mxnetlinux-cpu') {
ws('workspace/build-cpu') {
init_git()
docker_run('ubuntu_cpu','build_ubuntu_cpu', false)
Expand All @@ -91,7 +91,7 @@ try {
}

stage('MBCC Inference'){
node('mxnetlinux-cpu') {
node('restricted-mxnetlinux-cpu') {
ws('workspace/modelBackwardsCompat') {
init_git()
unpack_lib('cpu', mx_lib)
Expand All @@ -101,13 +101,13 @@ try {
}
}
} catch (caughtError) {
node("mxnetlinux-cpu") {
node("restricted-mxnetlinux-cpu") {
sh "echo caught ${caughtError}"
err = caughtError
currentBuild.result = "FAILURE"
}
} finally {
node("mxnetlinux-cpu") {
node("restricted-mxnetlinux-cpu") {
// Only send email if model backwards compat test failed
if (currentBuild.result == "FAILURE") {
emailext body: 'Nightly tests for model backwards compatibity on MXNet branch : ${BRANCH_NAME} failed. Please view the build at ${BUILD_URL}', replyTo: '${EMAIL}', subject: '[MODEL BACKWARDS COMPATIBILITY TEST FAILED] build ${BUILD_NUMBER}', to: '${EMAIL}'
Expand Down
4 changes: 2 additions & 2 deletions tests/nightly/model_backwards_compatibility_check/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@

# get the current mxnet version we are running on
mxnet_version = mx.__version__
model_bucket_name = 'mxnet-model-backwards-compatibility-models'
model_bucket_name = 'mxnet-ci-prod-backwards-compatibility-models'
piyushghai marked this conversation as resolved.
Show resolved Hide resolved
data_folder = 'mxnet-model-backwards-compatibility-data'
backslash = '/'
s3 = boto3.resource('s3')
Expand Down Expand Up @@ -207,4 +207,4 @@ def compare_versions(version1, version2):
'''
def normalize(v):
return [int(x) for x in re.sub(r'(\.0+)*$','', v).split(".")]
return cmp(normalize(version1), normalize(version2))
return cmp(normalize(version1), normalize(version2))