diff --git a/docs/source/developers/release.rst b/docs/source/developers/release.rst index 0263ce17a2c..c8b7e15a140 100644 --- a/docs/source/developers/release.rst +++ b/docs/source/developers/release.rst @@ -487,7 +487,39 @@ Be sure to go through on the following checklist: :class-title: sd-fs-5 :class-container: sd-shadow-md - TODO: Move from https://cwiki.apache.org/confluence/display/ARROW/Release+Management+Guide#ReleaseManagementGuide-UpdatingRpackages + To publish the R package on CRAN, there are a few steps we need to do first + in order to ensure that binaries for Windows and macOS are available to CRAN. + Jeroen Ooms maintains several projects that build C++ + dependencies for R packages for macOS and Windows. We test copies of these + same build scripts in our CI, and at release time, we need to send any + changes we have and update the versions/hashes upstream. + + When the release candidate is made, make draft pull requests to each + repository using the rc, updating the version and SHA, as well as any cmake + build changes from the corresponding files in apache/arrow. Jeroen may + merge these PRs before the release vote passes, build the binary artifacts, + and publish them in the right places so that we can do pre-submission checks + (see below). After the release candidate vote passes, update these PRs + to point to the official (non-rc) URL and mark them as ready for review. + Jeroen will merge, build the binary artifacts, and publish them in the + right places. See the + `packaging checklist `_. + for a precise list of pull requests that must be made prior to submission + to CRAN. + + Once these binary prerequisites have been satisfied, we can submit to CRAN. + Given the vagaries of the process, it is best if the R developers on the + project verify the CRAN-worthiness of the package before submitting. + Our CI systems give us some coverage for the things that CRAN checks, but + there are a couple of final tests we should do to confirm that the release + binaries will work and that everything runs on the same infrastructure that + CRAN has, which is difficult/impossible to emulate fully on Travis or with + Docker. For a precise list of checks, see the + `packaging checklist `_. + + Once all checks are clean, we submit to CRAN, which has a web form for + uploading packages. The release process requires email confirmation + from the R package maintainer, currently Neal Richardson. .. dropdown:: Update vcpkg port :animate: fade-in-slide-down diff --git a/r/.Rbuildignore b/r/.Rbuildignore index 2259aabe396..756b6faf116 100644 --- a/r/.Rbuildignore +++ b/r/.Rbuildignore @@ -30,3 +30,4 @@ STYLE.md ^cheatsheet$ ^revdep$ ^vignettes$ +^PACKAGING\.md$ diff --git a/r/PACKAGING.md b/r/PACKAGING.md new file mode 100644 index 00000000000..c2f53440652 --- /dev/null +++ b/r/PACKAGING.md @@ -0,0 +1,136 @@ + + + +# Packaging checklist for CRAN release + +For a high-level overview of the release process see the +[Apache Arrow Release Management Guide](https://arrow.apache.org/docs/developers/release.html#post-release-tasks). + +Before the release candidate is cut: + +- [ ] [Create a GitHub issue](https://github.com/apache/arrow/issues/new/) + entitled `[R] CRAN packaging checklist for version X.X.X` + and copy this checklist to the issue. +- [ ] Evaluate the status of any failing + [nightly tests and nightly packaging builds](http://crossbow.voltrondata.com). These checks + replicate most of the checks that CRAN runs, so we need them all to be passing + or to understand that the failures may (though won't necessarily) result in a rejection from CRAN. +- [ ] Check [current CRAN check results](https://cran.rstudio.org/web/checks/check_results_arrow.html) +- [ ] Ensure the contents of the README are accurate and up to date +- [ ] Run `urlchecker::url_check()` on the R directory at the release candidate + commit. Ignore any errors with badges as they will be removed in the CRAN release branch. +- [ ] [Polish NEWS](https://style.tidyverse.org/news.html#news-release) but do **not** update version numbers (this is done automatically later). +- [ ] For major releases, prepare tweet thread highlighting new features + +Wait for the release candidate to be cut: + +- [ ] Release candidate! +- [ ] Create a CRAN-release branch from the release candidate commit + +Make pull requests into the [autobrew](https://github.com/autobrew) and +[rtools-packages](https://github.com/r-windows/rtools-packages) repositories +used by the configure script on MacOS and Windows. These pull requests will +use the release candidate as the source. + +- [ ] Pull request to modify + [the apache-arrow autobrew formula]( https://github.com/autobrew/homebrew-core/blob/high-sierra/Formula/apache-arrow.rb) + to update the release version, SHA256 checksum of the release source file (which can be found in the same directory as the release source file), and any changes to dependencies and build steps that have changed in the + [copy of the formula we have of that formula in the Arrow repo](https://github.com/apache/arrow/blob/master/dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb) +- [ ] Pull request to modify + [the apache-arrow-static autobrew formula]( https://github.com/autobrew/homebrew-core/blob/master/Formula/apache-arrow-static.rb) + to update the version, SHA, and any changes to dependencies and build steps that have changed in the + [copy of the formula we have of that formula in the Arrow repo](https://github.com/apache/arrow/blob/master/dev/tasks/homebrew-formulae/autobrew/apache-arrow-static.rb) +- [ ] Pull request to modify the + [autobrew script](https://github.com/autobrew/scripts/blob/master/apache-arrow) + to include any additions made to + [r/tools/autobrew](https://github.com/apache/arrow/blob/master/r/tools/autobrew). +- [ ] Pull request to modify the + [RTools PKGBUILD script](https://github.com/r-windows/rtools-packages/blob/master/mingw-w64-arrow/PKGBUILD) + to reflect changes in + [ci/PKGBUILD](https://github.com/apache/arrow/blob/master/ci/scripts/PKGBUILD), + uncommenting the line that says "uncomment to test the rc". + +Prepare and check the .tar.gz that will be released to CRAN. + +- [ ] `git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d` +- [ ] Run `make build`. This copies Arrow C++ into tools/cpp, prunes some + unnecessary components, and runs `R CMD build` to generate the source tarball. + Because this will install the package, you will need to ensure that the version + of Arrow C++ available to the configure script is the same as the version + that is vendored into the R package (e.g., you may need to unset `ARROW_HOME`). +- [ ] `devtools::check_built("arrow_X.X.X.tar.gz")` locally +- [ ] Run reverse dependency checks. Currently this is a + [manual process](https://gist.github.com/paleolimbot/630fdab1e204d70fea97633d8fa15ccb); + however, in the future it may be a crossbow nightly job. + +Wait for the official release... + +- [ ] Release vote passed! +- [ ] If the release candidate commit updated, rebase the CRAN release branch + on that commit. +- [ ] Pick any commits that were made to master since the release commit that + were needed to fix CRAN-related submission issues identified in the above + steps. +- [ ] Remove badges from README.md +- [ ] Run `urlchecker::url_check()` on the R directory +- [ ] Create a PR entitled `WIP: [R] Verify CRAN release-10.0.1-rc0`. Add + a comment `@github-actions crossbow submit --group r` to run all R crossbow + jobs against the CRAN-specific release branch. +- [ ] Regenerate arrow_X.X.X.tar.gz (i.e., `make build`) + +Create new autobrew and r-windows PRs such that they use the *release* +instead of the *release candidate*; ensure linux binary packages are available: + +- [ ] PR into autobrew/homebrew-core (apache-arrow autobrew formula) +- [ ] PR into autobrew/homebrew-core (apache-arrow-static autobrew formula) +- [ ] PR into autobrew/scripts +- [ ] PR into r-windows/rtools-packages +- [ ] Ensure linux binaries are available in the artifactory: + https://apache.jfrog.io/ui/repos/tree/General/arrow/r + +Check binary Arrow C++ distributions specific to the R package: + +- [ ] Upload the .tar.gz to [win-builder](https://win-builder.r-project.org/upload.aspx) (r-devel only) + and confirm (with Neal, who will automatically receive an email about the results) that the check is clean. + This step cannot be completed before Jeroen has put the binaries in the MinGW repository, i.e. [here](https://ftp.opencpu.org/rtools/ucrt64/), [here](https://ftp.opencpu.org/rtools/mingw64/), and [here](https://ftp.opencpu.org/rtools/mingw32/). +- [ ] Upload the .tar.gz to [MacBuilder](https://mac.r-project.org/macbuilder/submit.html) + and confirm that the check is clean +- [ ] Check `install.packages("arrow_X.X.X.tar.gz")` on Ubuntu and ensure that the + hosted binaries are used +- [ ] `devtools::check_built("arrow_X.X.X.tar.gz")` locally one more time (for luck) + +Submit! + +- [ ] Upload arrow_X.X.X.tar.gz to the + [CRAN submit page](https://xmpalantir.wu.ac.at/cransubmit/) +- [ ] Confirm the submission email + +Wait for CRAN... + +- [ ] Accepted! +- [ ] Tag the tip of the CRAN-specific release branch +- [ ] Add a new line to the matrix in the [backwards compatability job](https://github.com/apache/arrow/blob/master/dev/tasks/r/github.linux.arrow.version.back.compat.yml) +- [ ] (patch releases only) Update the package version in `ci/scripts/PKGBUILD`, `dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb`, `r/DESCRIPTION`, and `r/NEWS.md` +- [ ] Update the packaging checklist template to reflect any new realities of the + packaging process. +- [ ] Wait for CRAN-hosted binaries on the + [CRAN package page](https://cran.r-project.org/package=arrow) to reflect the + new version +- [ ] Tweet!