design and implement phase2 and phase3 handoff (a.k.a. deploy addons)

[mikedanese]

Right now addon configuration can be generated from the cluster config object. We need a way to run it automatically after phase2 on the master completes. kube-up uses the kube-addon-manager.sh which is a mess which we should avoid. Open questions:

• Where are templates expanded? On the local machine or the master?
• Do we support upgrades? If so how?
• Is this a continuously running and reconciling process or does it only run once?

[errordeveloper]

Where are templates expanded?

I wonder whether this is really needed.

Looking at SkyDNS right now, it has the following:

1. DNS_REPLICAS and DNS_SERVER_IP, which are attributes in a config file
2. DNS_DOMAIN, which is passed as an argument

I am pretty sure that a config map can be used for 2, not quite sure about 1.

I'd guess a simple and flexible approach would be to use a git repo, and make updates optional through git mechanics. User could override the repo location, if they really have to set different parameters to add-ons they can fork and edit. However, I'm only familiar with SkyDNS add-on, and the DNS_SERVER_IP is something that I've had to care about, as the default (10.0.0.10), is too vulnerable to clash, so I'd rather have a separate discussion of that...

Do we support upgrades? If so how?

As I suggested, using a git repo may be a simple way to start. I've also whether user might want to freeze certain addons, and in that case they would just have to make sure to not commit any changes... Well, this now becomes a little separate project, some folks might like using such a thing for doing simple CI/CD stuff. Also, would be good to check the status of updates for daemon sets, last time I checked, it didn't work yet, although it does't sound too complicated to handle daemon sets in different fashion. I'm happy to think of a simple design/prototype for this, if the general idea of using git sounds plausible.

Is this a continuously running and reconciling process or does it only run once?

With the approach described above, it would have to run constantly or, at least periodically.

I am also thinking that some user might prefer to perform explicit updates to the set of addons, which may be a kind of thing they'd do with Ansible (or other such thing)... That's just for the record. I'd go with a single easy solution for the start.

[mikedanese]

It would be great if the configuration required was minimal enough to not require templating. I think for the MVP, we need to be able to apply addons to the control-plane once (probably at the end of the deployment process). Versioning through git also sounds good. We should investigate https://github.com/kubernetes/helm and see if it has the capabilities we require from an addon applier. It would be great not to have to roll our own. Explicit updates sound good for the initial implementation.

[errordeveloper]

Yes, I was wondering if Helm would be appropriate to provide out of the box.

[errordeveloper]

See helm/helm#838 (comment) for details of my investigation so far, basically I didn't get very far with it. Hacking a PoC version of Helm that would work with kube-system namespaces seems doable, but let's see if Helm maintainers are planning to work on this any time soon.

[errordeveloper]

Ok, actually helm/helm#852 fixes the issue, I'll carry on trying out what it takes to actually use Helm for managing addons.

[errordeveloper]

Where are templates expanded? On the local machine or the master?

Helm solves templating question, and I'd be happy convert addons to charts with the obvious parameters (ones that are currently substitute by kube-up.sh and image versions that are currently bumped manually).

I've converted dashboard addon to illustrate how this might look like.

Do we support upgrades? If so how?

Helm currently doesn't have an upgrade mechanism helm/helm#690, but we haven't decided if that is required, so may be it's okay to wait for it to appear in Helm eventually, as it's most likely will.

Is this a continuously running and reconciling process or does it only run once?

Regarding this item, I suppose Helm would provide enough flexibility to how addons are managed, except for upgrades being an outstanding feature...

[errordeveloper]

@mikedanese good catch on the title, I though something was odd with it 👍

[mikedanese]

Thanks for exploring this. In your opinion is it to early to move to helm or should we go for it? Would it be easy to selectively enable addons? https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase3/Kconfig

[errordeveloper]

I haven't used Helm super extensively to judge how well it works in the
long term, but appears to be fairly simple right now. The Helm team are on
this usecase, so that's good. Regarding selecting add-ons, we can do a
simple loop over selected ones in Ansible, and Helm would take care of
where the definitions are located and do the template rendering in a
non-adhoc fashion. I am not hugely impressed with usability of Go
templates, but it has a linter that can be improved.

On Wed, 22 Jun 2016, 19:41 Mike Danese, [email protected] wrote:

Thanks for exploring this. In your opinion is it to early to move to helm
or should we go for it? Would it be easy to selectively enable addons?
https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase3/Kconfig

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#126 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAPWS0EPh_3wgXwrXJ8LQ7yUeb8Tj7_Dks5qOYHOgaJpZM4IzX_Z
.

[fejta-bot]

Issues go stale after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle rotten
/remove-lifecycle stale

[errordeveloper]

/close