-
Notifications
You must be signed in to change notification settings - Fork 535
WIP Upgrade to rebar3 3.18, with improvements in packaging and release assembly #1097
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
hmmr
wants to merge
49
commits into
basho:develop-3.0
from
TI-Tokyo:hmmr/develop-3.0/upgrade-to-rebar3.17
Closed
Changes from 13 commits
Commits
Show all changes
49 commits
Select commit
Hold shift + click to select a range
5ddfd28
nonexistent target ballclean
4ec03f3
determine otp version from erlang:system_info(otp_release)
df64555
upgrade rebar3 to 3.17 plus git revlist fix
73f12e9
upgrade rebar3_cuttlefish (to a version that supports fqdn data type)
3db7582
avoid copying .git dirs when preparing rel- targets
hmmr 6976d5e
improvements in deb packaging (compliance with current standards, etc)
b5951f8
clean up after dropping yokozuna
10598c5
improvements in rpm packaging
d83d0ae
enable packaging for Amazon Linux 2
cb67886
osx package
debd415
cleaner su in launcher script
8b95c99
refactor and repair release assembly
9cb62b8
upgrade rebar3 to 3.18
88c76ff
rm node_package relic
9f25dab
alpine packaging groundwork
b052781
launcher without su, just for alpine
8009ff2
copy APKBUILD, with instruction on how to use it
9149d75
packaging for freebsd
2fa2d8b
use https:// when fetching github deps
hmmr 48e829a
tmp repoint, bump lager_syslog
hmmr cfcacbc
Merge branch 'hmmr/develop-3.0/upgrade-to-rebar3.17' of github.com:TI…
hmmr ffb83b6
update rel/pkg/alpine/README
hmmr 918f17b
more dependency bumps
hmmr e6f64bf
pre-create ./log in 'rel' release (fixes silent fail on first riak st…
hmmr bb10751
read release version from git tag
hmmr 6ab6b0b
rm a leftover in rel/pkg/rpm
hmmr 1ce0262
deprecate, warn about hyphenated forms of riak-admin
hmmr 937e08d
pid file for local release is useless
c74a556
change to rebar3_cuttlefish-0.2.1 (0.2.0 with rebar3.18 API fix)
3628c74
riak_not_running is actually a template
5c0009e
let relx handle pid files (unbroken in rebar3_cuttlefish-0.2.1)
58b063a
rm leftover debian/riak.init
f75df62
rebar3.org recommends starting riak with 'foreground'
a50898f
tmp point rebar3_cuttlefish to a branch
4a3f0ff
missing items in debian/control BuildDepends
3140383
debian/Makefile: don't do make clean when nothing has been built yet
de4b6e9
deb/Makefile: link _build/default/{libs,plugins} to avoid fetching do…
36d6970
export RUNNER_LOG_DIR to unbreak cuttlefish-generated scripts
a6b196a
move hooks into own dir under rel/files
b03a592
why prepend riak- to PKG_ID?
f776cd7
symlnk deps from _build/default/lib/* into tree snapshot in rel/pkg/o…
8647c98
try explicit config/vm.args, to unbreak relx internode ops
323453d
amend de4b6e9a to only symlink libs (cuttlefish plugin fails unless r…
231d6f1
Merge branch 'hmmr/develop-3.0/fbsdng-support' into hmmr/develop-3.0/…
2afabb2
point deps to develop-3.0 branches
f0dc94c
tmp remove rebar.lock while deps are floated
332f71f
typo in rebar.config
e0b92c3
with rebar3_cuttlefish unbroken, undo hyphenated command routing
hmmr 57314bd
make use of cf_config in riak-chkconfig, again
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file not shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| .deps/ | ||
| .libs/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,57 +1,87 @@ | ||
| #!/bin/bash | ||
|
|
||
| RUNNER_GEN_DIR="${RUNNER_GEN_DIR:-{{ platform_gen_dir }}}" | ||
| RELEASE_ROOT_DIR="${RELEASE_ROOT_DIR:-{{ runner_base_dir }}}" | ||
| PID_DIR={{pid_dir}} | ||
| COMMAND={{platform_bin_dir}}/riak | ||
| DEBUG_COMMAND={{platform_bin_dir}}/riak-debug | ||
| RUNNER_LOG_DIR={{platform_log_dir}} | ||
| RELX_CONFIG_PATH=${RUNNER_GEN_DIR}/sys.config | ||
| VMARGS_PATH=${RUNNER_GEN_DIR}/vm.args | ||
| RUNNER_GEN_DIR={{platform_gen_dir}} | ||
| RELX_RIAK={{platform_bin_dir}}/riak | ||
| export PID_DIR={{pid_dir}} | ||
| export PIPE_DIR={{pipe_dir}}/ # terminating / (relx treats it as a prefix) | ||
|
|
||
| mkdir -p $PID_DIR $PIPE_DIR | ||
| chown riak:riak $PID_DIR $PIPE_DIR | ||
|
|
||
| # On first running the sys.config and vm.args will not be a link | ||
| # as cfconfig has not yet been run as a pre_start hook. If there's no | ||
| # link use the default for now | ||
| if [ ! -L $RELX_CONFIG_PATH ]; then | ||
| cp $RELEASE_ROOT_DIR/releases/{{release_version}}/sys.config $RELX_CONFIG_PATH | ||
| fi | ||
| if [ ! -L $VMARGS_PATH ]; then | ||
| cp $RELEASE_ROOT_DIR/releases/{{release_version}}/vm.args $VMARGS_PATH | ||
| fi | ||
| function write_pid_file { # relx doesn't seem to write the pid file, does it? | ||
| local c=3 | ||
| sleep 1 | ||
| while [[ $c -ne 0 ]]; do | ||
| if [ "`maybe_su ${RELX_RIAK} ping`" = "pong" ]; then | ||
| maybe_su ${RELX_RIAK} pid >$PID_DIR/riak.pid | ||
| # systemd wants it to be owned by root, so: | ||
| chown root:root $PID_DIR/riak.pid 2&>1 >/dev/null \ | ||
| || true # if we are run as non-root, just never mind | ||
| break | ||
| fi | ||
| sleep 2 | ||
| c=$(($c - 1)) | ||
| done | ||
| } | ||
|
|
||
| function delete_pid_file { | ||
| rm -f $PID_DIR/riak.pid | ||
| } | ||
|
|
||
| # When running as a service, running as riak not as root, and systemd has created PID folder | ||
| if [[ $EUID -ne 0 ]]; then | ||
| case "$1" in | ||
| start|console|foreground) | ||
| RELX_CONFIG_PATH=${RELX_CONFIG_PATH} VMARGS_PATH=${VMARGS_PATH} RUNNER_LOG_DIR=${RUNNER_LOG_DIR} PIPE_DIR=${PIPE_DIR} ${COMMAND} ${*} -pa {{platform_lib_dir}}/patches | ||
| ;; | ||
| *) | ||
| RELX_CONFIG_PATH=${RELX_CONFIG_PATH} VMARGS_PATH=${VMARGS_PATH} RUNNER_LOG_DIR=${RUNNER_LOG_DIR} PIPE_DIR=${PIPE_DIR} ${COMMAND} ${*} | ||
| ;; | ||
| esac | ||
| # centos7-based distros have a su that contacts pam and prints the "Last logged in" message | ||
| if [ "`cat /etc/redhat-release 2>&1`" = "CentOS Stream release 8" ] || | ||
| [ "`cat /etc/system-release 2>&1`" = "Amazon Linux release 2 (Karoo)" ]; then | ||
| SU="su --session-command" | ||
| else | ||
| # In this case we're running sudo riak - so have root access, but cannot rely | ||
| # systemd having created the PID dir, and need to sudo to the riak user | ||
| if [ ! -d $PID_DIR ]; then | ||
| mkdir $PID_DIR | ||
| chown riak:riak $PID_DIR | ||
| fi | ||
| case "$1" in | ||
| start|console|foreground) | ||
| su - riak -c "NODETOOL_NODE_PREFIX=${NODETOOL_NODE_PREFIX} RELX_CONFIG_PATH=${RELX_CONFIG_PATH} VMARGS_PATH=${VMARGS_PATH} RUNNER_LOG_DIR=${RUNNER_LOG_DIR} PIPE_DIR=${PIPE_DIR} ${COMMAND} ${*} -pa {{platform_lib_dir}}/patches" | ||
| ;; | ||
| debug) | ||
| # Drop the "debug" from the args as we're going to directly call riak-debug now | ||
| shift | ||
| # Debug may fail if run via relx script due to use of relative start location, and also need for root access | ||
| NODETOOL_NODE_PREFIX=${NODETOOL_NODE_PREFIX} RELX_CONFIG_PATH=${RELX_CONFIG_PATH} VMARGS_PATH=${VMARGS_PATH} RUNNER_LOG_DIR=${RUNNER_LOG_DIR} PIPE_DIR=${PIPE_DIR} ${DEBUG_COMMAND} ${*} | ||
| ;; | ||
| *) | ||
| ESCAPED_ARGS=`echo "$@" | sed -e 's/\([\\\(\\\){}"\x27]\)/\\\\\1/g'` | ||
| su - riak -c "NODETOOL_NODE_PREFIX=${NODETOOL_NODE_PREFIX} RELX_CONFIG_PATH=${RELX_CONFIG_PATH} VMARGS_PATH=${VMARGS_PATH} RUNNER_LOG_DIR=${RUNNER_LOG_DIR} PIPE_DIR=${PIPE_DIR} ${COMMAND} ${ESCAPED_ARGS}" | ||
| ;; | ||
| esac | ||
| SU=su | ||
| fi | ||
|
|
||
| function maybe_su { | ||
| if [[ $EUID -ne 0 ]]; then | ||
| $* | ||
| else | ||
| # if we are executing an admin command that spins up a | ||
| # (hidden) node to then execute custom erlang code via -eval, | ||
| # we need to cd to a dir containing the erlang cookie | ||
| # (previously implicitly done by su -, which option we have | ||
| # removed in order to allow any env vars to be available for | ||
| # the ultimate invocation of riak/riak-cs/stanchion) | ||
| cd {{platform_base_dir}} | ||
| $SU riak -c "$*" | ||
| fi | ||
| } | ||
|
|
||
| case "$1" in | ||
| start) | ||
| maybe_su $RELX_RIAK $* -pa {{platform_patch_dir}} \ | ||
| && write_pid_file | ||
| test -r $PID_DIR/riak.pid && exit 0 | ||
| ;; | ||
| console|foreground) | ||
| maybe_su $RELX_RIAK $* -pa {{platform_patch_dir}} | ||
| ;; | ||
| stop) | ||
| maybe_su $RELX_RIAK $* \ | ||
| && delete_pid_file | ||
| ;; | ||
| admin) | ||
| shift | ||
| maybe_su `which riak-admin` $* | ||
| ;; | ||
| chkconfig) | ||
| shift | ||
| maybe_su `which riak-chkconfig` $* | ||
| ;; | ||
| repl) | ||
| shift | ||
| maybe_su `which riak-repl` $* | ||
| ;; | ||
| debug) | ||
| shift | ||
| maybe_su `which riak-debug` $* | ||
| ;; | ||
| *) | ||
| ESCAPED_ARGS=`echo "$@" | sed -e 's/\([\\\(\\\){}"\x27]\)/\\\\\1/g'` | ||
| maybe_su $RELX_RIAK $ESCAPED_ARGS | ||
| ;; | ||
| esac | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does each command need to be present in this case - is this not already handled in the extended start script as extensions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's an effort to make hyphenated and non-hyphenated forms equivalent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think it works as expected, and I'm not even sure that resurrecting the hyphenated commands is warranted - what purpose does it serve? It's been a couple of years since we moved to start script extensions. You now have to re-insert a bunch of boilerplate environment stuff at the top of the script files to make them work, whereas that was all inherited with from the root start script with extensions.
My opinion is that it's a backwards step, unless there's a compelling reason that I'm not aware of / considering.
The tidy up across the rest of the release config is very decent though, and much needed. Some of the deps currently reference TI forks, which would need to target basho, but guess that would be fixed up before everything gets merged in.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
To deal with the easier part first: re-pinning deps to TI-Tokyo repos has a single purpose: pull in cuttlefish and rebar3_cuttlefish, not so much for the fqdn data type in cuttlefish but rather for several commits in the plugin that make it compatible with rebar3 3.18 (by ioolkos, see this one and thereabout), synchronize its internal install_update_escript with the "upstream" version of it in relx, and last but not least, this little gem.
I note that the fqdn-type patch has already been merged in basho/cuttlefish, but there is no tag. If you do the tagging, I will then prepare the necessary PRs to bump cuttlefish version where necessary (it is an explicit dependency in bitcask, clique, eleveldb and riak_sysmon), and collect a patchset for another PR in rebar3_cuttlefish.
Regarding the phasing out of hyphenated forms, I have no strong opinions and will gladly follow the party line (although I would invite @Bob-The-Marauder to this discussion, who has been in the business of dealing with customers for some time).
For the record, my understanding of the underlying machinery is as follows:
riakscript generated by relx and/or cuttlefish provides basic commands like start, stop, console, pid, ping, rpc. It is how riak is started/stopped in riak_test setting. It is not under our control.riak, the one created from the template in rel/files, has a distinct and specific purpose:suas appropriate, and is therefore installed in PATH when riak is installed from a package;riak-adminetc when called asriak admin.I saw this last item precisely as a way to enable non-hyphenated forms (if we remove riak-admin and its hyphenated friends from PATH, the move to hide hyphenated forms will be complete, from the user standpoint). Apparently, by "move to start script extensions" you mean to do it differently -- perhaps by injecting code in the script produced by relx? If so, I would be happy to rework my contribution accordingly, but I didn't see how relx can be massaged to achieve that.