No Downtime Deployment of Packs

[nmaludy]

Background

One of the goals i have for our StackStorm deployment is to implement Continuous Delivery (CD) of our custom packs without having to schedule downtime of our StackStorm servers.

Requirements

• Upgrades can be performed without taking downtime
• No race conditions exist that could cause errors of ongoing workings during a deployment
• External systems, that may be polling for status/completion of ongoing workflows, should not be affected
• Packs may need to deploy/update virtualenvs
• Pack configs may need to be loaded
• Key/value pairs may need to be added to the datastore
• Custom content may need to be installed on the box (deploy additional non-pack content)
• Custom scripts may need to be run on the box (install system level dependencies)

Conclusion

This issue is to simply start the conversation and brainstorm ideas of how to implement such a feature/process.

[cognifloyd]

I have one pack that depends on a package that introduces a dependency on pandas... Building that virtualenv takes forever. It would be marvelous to be able to update that pack and virtualenv without downtime.

I would guess that virtualenvs could be rebuilt by building them in one folder, and then moving the virtualenv into place once built. Or maybe, copy the existing virtualenv to a new folder, update it, then move it into place (moving the old out of the way at the same time). Perhaps something similar could be done for pack install.

Using https://github.com/EncoreTechnologies/ansible-stackstorm to install packs, it uninstalls and reinstalls the pack because there's no good way to idempotently install/update a pack. That will probably have to be addressed to make this work. This is issue #3052, but probably includes resolving these issues as well: #3191, #3313, #924, #3143, #3016, #3749, #3660, #3585, #3336

[cognifloyd]

Or, if building a virtualenv in an alternate folder causes issues... ST2 could start using a chroot or a FS overlay (I'm thinking some container-like thing that overlays on top of the FS - I think systemd can do that on the fly iirc) for building virtualenvs and installing packs, only swapping them into place once complete.

[nmaludy]

In-place Deployments

From my previous experiences i believe this deployment strategy will be the hardest to get right technically. It will involve a lot of "transactional" type operations which are non-trivial to implement especially on the system level.

Blue/Green Deployments

I think this has the best chance of being a workable solution. Basically, front the StackStorm instance with a load-balancer, stand up another StackStorm instance with the new code deployed and then flip a load-balancer over to the new instance.

The trick here is, how do you allow external systems, that may be interacting with the st2 API to still work without issue? An external system may be waiting on the result of a currently running action and if the load-balancer flips over to a new instance and a new query is made, that execution ID doesn't exist.

[cognifloyd]

Perhaps the In-place Deployment is not as clean as Blue/Green, but it would still make sense for pack installs/upgrades to work better in this case, even if it requires a moment of down time for the install transaction. Perhaps it would require waiting for no pack actions to run, or temporarily 'pausing' running things (preventing them from starting until after the pack upgrade) for a particular pack while the transaction to swap old/new pack install (pack files, db entries, and virtualenv) is running.

Eventually I'd love to get to the point where I can do Blue/Green Deployments on anything. Until then, I'm using ansible and ST2 to migrate away from our snowflake infra to a more predictable/automatable infra. Until a lot more gets pulled out of the snowflake, Blue/Green isn't an option for me. So, making In-place work better would be great. :)

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[nmaludy]

Still an issue.

[stale]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically marking is as stale. If this issue is not relevant or applicable anymore (problem has been fixed in a new version or similar), please close the issue or let us know so we can close it. On the contrary, if the issue is still relevant, there is nothing you need to do, but if you have any additional details or context which would help us when working on this issue, please include it as a comment to this issue.

[nmaludy]

Still an issue.