Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Beta build #16411

Merged
merged 3 commits into from
Oct 10, 2019
Merged

Beta build #16411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
59db9de
remove unused jenkinsfile
aaronmarkham Oct 9, 2019
863abc6
add a pipeline that publishes to staging beta site
aaronmarkham Oct 9, 2019
57185b3
add staging site tar file generation; fix outputs
aaronmarkham Oct 10, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions ci/docker/runtime_functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1832,12 +1832,12 @@ build_docs() {
popd
}

build_docs_small() {
build_docs_beta() {
pushd docs/_build
tar -xzf jekyll-artifacts.tgz
api_folder='html/api'
mkdir -p $api_folder/python/docs && tar -xzf python-artifacts.tgz --directory $api_folder/python/docs
# The folder to be published is now in /docs/_build/html
GZIP=-9 tar -zcvf beta_website.tgz -C html .
popd
}

Expand Down
43 changes: 41 additions & 2 deletions ci/jenkins/Jenkins_steps.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1416,11 +1416,11 @@ def should_pack_website() {
return true
}
} else {
return true
return true
}
return false
}

// Each of the docs_{lang} functions will build the docs...
// Stashing is only needed for master for website publishing or for testing "new_"

Expand Down Expand Up @@ -1604,6 +1604,27 @@ def docs_prepare() {
}


def docs_prepare_beta() {
return ['Prepare for publication to the staging website': {
node(NODE_LINUX_CPU) {
ws('workspace/docs') {
timeout(time: max_time, unit: 'MINUTES') {
utils.init_git()

unstash 'jekyll-artifacts'
unstash 'python-artifacts'

utils.docker_run('ubuntu_cpu_jekyll', 'build_docs_beta', false)

// archive so the publish pipeline can access the artifact
archiveArtifacts 'docs/_build/beta_website.tgz'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where is this being prepared ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did I miss something?
Job calls ci/jenkins/Jenkinsfile_website_beta
--> unix_cpu_lite --> binary
--> docs_jekyll --> jekyll artifacts (stash)
--> docs_python --> python api artifacts (stash)
--> docs_prepare_beta (unstash both) --> archive beta_website.tgz
--> docs_publish_beta (using archive)

I did forget to update the filename in the publish job, so this was a worthwhile exercise 🏃 😄

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are running utils.docker_run('ubuntu_cpu_jekyll', 'build_docs', false)
check runtime_function.sh:1819, the doc/_build/beta_website.tgz is not going to be created by magic 😄
You need to create a new runtime function, build_docs_beta for example that would do something like this:

build_docs_beta() {
    pushd docs/_build
    tar -xzf jekyll-artifacts.tgz
    api_folder='html/api'

    # Python has it's own landing page/site so we don't put it in /docs/api
    mkdir -p $api_folder/python/docs && tar -xzf python-artifacts.tgz --directory $api_folder/python/docs
    GZIP=-9 tar -zcvf beta_website.tgz -C html .
    popd
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But please let's try to not diverge between bets and prod code as much as possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But please let's try to not diverge between bets and prod code as much as possible.

This does have the drawback of not rendering most of the other microsites, but then we get a much faster way to stage updates. How about after most of the restructuring is done on the Python API we go back and add in the rest of the APIs so the staging run has everything in it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good! :)

}
}
}
}]
}


def docs_archive() {
return ['Archive the full website': {
node(NODE_LINUX_CPU) {
Expand Down Expand Up @@ -1639,6 +1660,24 @@ def docs_publish() {
}


// This is for the beta website
def docs_publish_beta() {
return ['Publish the beta website to staging': {
node(NODE_LINUX_CPU) {
ws('workspace/docs') {
timeout(time: max_time, unit: 'MINUTES') {
try {
build 'restricted-website-publish-master-beta'
}
catch (Exception e) {
println(e.getMessage())
}
}
}
}
}]
}


def misc_asan_cpu() {
return ['CPU ASAN': {
Expand Down
23 changes: 19 additions & 4 deletions ci/jenkins/Jenkinsfile_website → ci/jenkins/Jenkinsfile_website_beta
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,24 +19,39 @@
//
// Jenkins pipeline
// See documents at https://jenkins.io/doc/book/pipeline/jenkinsfile/
// This pipeline will publish to https://mxnet-beta.staged.apache.org/

// timeout in minutes
max_time = 180

node('utility') {
node('restricted-utility') {
// Loading the utilities requires a node context unfortunately
checkout scm
utils = load('ci/Jenkinsfile_utils.groovy')
custom_steps = load('ci/jenkins/Jenkins_steps.groovy')
}
utils.assign_node_labels(utility: 'utility', linux_cpu: 'mxnetlinux-cpu')

utils.assign_node_labels(utility: 'restricted-utility', linux_cpu: 'restricted-mxnetlinux-cpu', linux_gpu: 'restricted-mxnetlinux-gpu', linux_gpu_p3: 'restricted-mxnetlinux-gpu-p3', windows_cpu: 'restricted-mxnetwindows-cpu', windows_gpu: 'restricted-mxnetwindows-gpu')

utils.main_wrapper(
core_logic: {
utils.parallel_stage('Placeholder', [
// Do nothing
utils.parallel_stage('Build', [
custom_steps.compile_unix_lite()
])

utils.parallel_stage('Build Docs', [
// Only building a subset of the docs for previewing on staging
custom_steps.docs_jekyll(),
custom_steps.docs_python()
])

utils.parallel_stage('Prepare', [
custom_steps.docs_prepare_beta()
])

utils.parallel_stage('Publish', [
custom_steps.docs_publish_beta()
])
}
,
failure_handler: {
Expand Down