Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upgrade to kubernetes 1.11 #473

Merged
merged 36 commits into from
Aug 7, 2018
Merged

Upgrade to kubernetes 1.11 #473

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
8c7b9b6
ConfigureKubelet: invert --read-only-port for new default
Jul 4, 2018
9d8dd2a
remove KUBE_DNS_ARGS, handled by kubeadm
Jul 4, 2018
88e5963
fix ConfigureDNS to patch coredns deployment
Jul 4, 2018
725058d
fix specs
Jul 4, 2018
5eb4980
fix etcd version check
Jul 4, 2018
1cf8599
update kube version to 1.11
Jul 4, 2018
b6c5d0b
update etcd to 3.2
Jul 4, 2018
1fe6a34
log kubeadm config at debug level
Jul 4, 2018
41f1f97
update kubeadm config for 1.11
Jul 4, 2018
b636d87
remove broken etcd_healthy check
Jul 4, 2018
4c8b329
change upgrade-kubeadm to download binary without updating system kub…
Jul 4, 2018
52b41c1
add 1.3 worker migration for running kubeadm upgrade node config
Jul 4, 2018
3398aea
fix kubeadm config spec
Jul 4, 2018
74a3f5c
rubofix
Jul 4, 2018
1c738b0
Merge branch 'master' into feature/kubernetes-1.11
Jul 25, 2018
a17646b
install kubeadm for upgrades at /usr/local/bin/pharos-kubeadm-*
Jul 25, 2018
16a3ba8
HostConfigurer#kubelet_args for centos --cgroup-driver
Jul 25, 2018
3b2ca3e
workaround systemd unit EnvironmentFile= KUBELET_EXTRA_ARGS override
Jul 25, 2018
04ff097
force kubeadm reset prompt
Jul 25, 2018
e2e0679
refactor configure_internal_etcd
Jul 30, 2018
5852fc8
rubofix
Jul 30, 2018
afeddca
fix missing ETCD_VERSION=3.2.18
Jul 30, 2018
28cb028
Merge branch 'master' into feature/kubernetes-1.11
jakolehm Aug 3, 2018
da369bb
cri-o 1.11.1
jakolehm Aug 3, 2018
488a074
Merge branch 'feature/kubernetes-1.11' of github.com:kontena/pharos-c…
jakolehm Aug 3, 2018
61157a4
patch coredns image
jakolehm Aug 3, 2018
1fd400f
coredns version
jakolehm Aug 3, 2018
9cd4b66
add gpg key in upgrade-kubeadm
jakolehm Aug 6, 2018
2849378
fix configure dns spec
jakolehm Aug 6, 2018
704b384
more specs
jakolehm Aug 6, 2018
23d83b7
kubernetes 1.11.1
jakolehm Aug 6, 2018
02150f9
rescue only KubeException
jakolehm Aug 6, 2018
467bc7a
specs
jakolehm Aug 7, 2018
75e1463
fix coredns podAntiAffinity to use old k8s-app=kube-dns label
Aug 7, 2018
c682ce5
rescue StandardError (fixes openssl errors leaking)
jakolehm Aug 7, 2018
dcdbda1
reset: cleanup pharos-kubeadm-* used for ugprades
Aug 7, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions lib/pharos/configuration/host.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,8 @@ def kubelet_args(local_only: false, cloud_provider: nil)
args << "--hostname-override=#{hostname}"
end

args += configurer(nil).kubelet_args

args
end

Expand Down
5 changes: 5 additions & 0 deletions lib/pharos/host/configurer.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,11 @@ def configure_cfssl
abstract_method!
end

# @return [Array<String>]
def kubelet_args
[]
end

# @param args [Hash]
def ensure_kubelet(args) # rubocop:disable Lint/UnusedMethodArgument
abstract_method!
Expand Down
5 changes: 5 additions & 0 deletions lib/pharos/host/el7/el7.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ def configure_cfssl
)
end

# @return [Array<String>]
def kubelet_args
['--cgroup-driver=systemd']
end

def configure_container_runtime
if docker?
exec_script(
Expand Down
2 changes: 1 addition & 1 deletion lib/pharos/host/el7/scripts/ensure-kubelet.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ mkdir -p /etc/systemd/system/kubelet.service.d
[Service]
ExecStartPre=-/sbin/swapoff -a
ExecStart=
ExecStart=/usr/bin/kubelet ${KUBELET_ARGS} --cgroup-driver=systemd --pod-infra-container-image=${IMAGE_REPO}/pause-${ARCH}:3.1
ExecStart=/usr/bin/kubelet ${KUBELET_ARGS} --pod-infra-container-image=${IMAGE_REPO}/pause-${ARCH}:3.1
EOF

yum_install_with_lock "kubelet" $KUBE_VERSION
Expand Down
3 changes: 3 additions & 0 deletions lib/pharos/host/el7/scripts/install-kube-packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@ yum_install_with_lock "kubelet" $KUBE_VERSION
yum_install_with_lock "kubectl" $KUBE_VERSION
yum_install_with_lock "kubeadm" $KUBE_VERSION

# use KUBELET_EXTRA_ARGS from /etc/systemd/system/kubelet.service.d/11-pharos.conf instead
sed -i 's/^KUBELET_EXTRA_ARGS=/#\0/' /etc/sysconfig/kubelet

if systemctl is-active --quiet rpcbind; then
systemctl stop rpcbind
systemctl disable rpcbind
Expand Down
5 changes: 3 additions & 2 deletions lib/pharos/host/el7/scripts/reset.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ if systemctl is-active --quiet crio ; then
systemctl disable crio
fi

kubeadm reset
kubeadm reset --force

yum remove -y kubeadm kubelet kubectl docker

Expand All @@ -39,7 +39,8 @@ sudo rm -rf /etc/kubernetes \
/usr/local/lib/cri-o-runc \
/usr/local/bin/skopeo \
/usr/local/bin/runc \
/usr/local/bin/crictl
/usr/local/bin/crictl \
/usr/local/bin/pharos-kubeadm-*

systemctl daemon-reload
systemctl reset-failed
15 changes: 10 additions & 5 deletions lib/pharos/host/el7/scripts/upgrade-kubeadm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,16 @@

set -ex

if [ $(kubeadm version -o short) = "v${VERSION}" ]; then
if [ -x /usr/local/bin/pharos-kubeadm-${VERSION} ]; then
exit
fi

versionlock="/etc/yum/pluginconf.d/versionlock.list"
linefromfile "^0:kubeadm-" $versionlock
yum install -y "kubeadm-${VERSION}"
lineinfile "^0:kubeadm-" "0:kubeadm-${VERSION}-0.*" $versionlock
BIN_URL="https://dl.bintray.com/kontena/pharos-bin/kube/${VERSION}/kubeadm-${ARCH}.gz"

curl -fsSL https://bintray.com/user/downloadSubjectPublicKey?username=bintray | gpg --import
curl -fsSL $BIN_URL -o /tmp/kubeadm.gz
curl -fsSL "${BIN_URL}.asc" -o /tmp/kubeadm.gz.asc
gpg --verify /tmp/kubeadm.gz.asc /tmp/kubeadm.gz
gunzip /tmp/kubeadm.gz
install -o root -g root -m 0755 -T /tmp/kubeadm /usr/local/bin/pharos-kubeadm-${VERSION}
rm /tmp/kubeadm /tmp/kubeadm.gz.asc
3 changes: 3 additions & 0 deletions lib/pharos/host/ubuntu/scripts/install-kube-packages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ apt-mark unhold kubelet kubectl kubeadm
apt-get install -y kubelet=${KUBE_VERSION}-00 kubectl=${KUBE_VERSION}-00 kubeadm=${KUBEADM_VERSION}-00
apt-mark hold kubelet kubectl kubeadm

# use KUBELET_EXTRA_ARGS from /etc/systemd/system/kubelet.service.d/11-pharos.conf instead
sed -i 's/^KUBELET_EXTRA_ARGS=/#\0/' /etc/default/kubelet
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe ConfigureKubelet should be writing the /etc/default/kubelet instead, but that's annoying since it varies by OS.


if ! dpkg -s nfs-common > /dev/null; then
systemctl mask rpcbind
apt-get install -y nfs-common
Expand Down
5 changes: 3 additions & 2 deletions lib/pharos/host/ubuntu/scripts/reset.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ if systemctl is-active --quiet crio ; then
systemctl disable crio
fi

kubeadm reset
kubeadm reset --force

apt-get purge -y --allow-change-held-packages --purge kubeadm kubelet kubectl docker.io
apt-get autoremove -y
Expand All @@ -39,7 +39,8 @@ rm -rf /etc/kubernetes \
/usr/local/lib/cri-o-runc \
/usr/local/bin/skopeo \
/usr/local/bin/runc \
/usr/local/bin/crictl
/usr/local/bin/crictl \
/usr/local/bin/pharos-kubeadm-*

systemctl daemon-reload
systemctl reset-failed
16 changes: 10 additions & 6 deletions lib/pharos/host/ubuntu/scripts/upgrade-kubeadm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,16 @@

set -ex

if [ $(kubeadm version -o short) = "v${VERSION}" ]; then
if [ -x /usr/local/bin/pharos-kubeadm-${VERSION} ]; then
exit
fi

cd /tmp
export DEBIAN_FRONTEND=noninteractive
apt-get download kubeadm=${VERSION}-00
dpkg -i --ignore-depends=kubelet kubeadm_${VERSION}*.deb
rm -f kubeadm_${VERSION}*.deb
BIN_URL="https://dl.bintray.com/kontena/pharos-bin/kube/${VERSION}/kubeadm-${ARCH}.gz"

curl -fsSL https://bintray.com/user/downloadSubjectPublicKey?username=bintray | gpg --import
curl -fsSL $BIN_URL -o /tmp/kubeadm.gz
curl -fsSL "${BIN_URL}.asc" -o /tmp/kubeadm.gz.asc
gpg --verify /tmp/kubeadm.gz.asc /tmp/kubeadm.gz
gunzip /tmp/kubeadm.gz
install -o root -g root -m 0755 -T /tmp/kubeadm /usr/local/bin/pharos-kubeadm-${VERSION}
rm /tmp/kubeadm /tmp/kubeadm.gz.asc
58 changes: 29 additions & 29 deletions lib/pharos/kubeadm.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,12 @@ def initialize(config, host)
# @return [Hash]
def generate_config
config = {
'apiVersion' => 'kubeadm.k8s.io/v1alpha1',
'apiVersion' => 'kubeadm.k8s.io/v1alpha2',
'kind' => 'MasterConfiguration',
'nodeName' => @host.hostname,

'nodeRegistration' => {
'name' => @host.hostname
},
'kubernetesVersion' => Pharos::KUBE_VERSION,
'imageRepository' => @config.image_repository,
'api' => {
Expand All @@ -35,24 +38,25 @@ def generate_config
'serviceSubnet' => @config.network.service_cidr,
'podSubnet' => @config.network.pod_network_cidr
},
'apiServerExtraArgs' => {},
'controllerManagerExtraArgs' => {
'horizontal-pod-autoscaler-use-rest-clients' => 'true'
},
'noTaintMaster' => !master_taint?
}
}

if @host.container_runtime == 'cri-o'
config['criSocket'] = '/var/run/crio/crio.sock'
unless master_taint?
config['nodeRegistration']['taints'] = []
end

config['apiServerExtraArgs'] = {
'apiserver-count' => @config.master_hosts.size.to_s
}
if @host.container_runtime == 'cri-o'
config['nodeRegistration']['criSocket'] = '/var/run/crio/crio.sock'
end

if @config.cloud && @config.cloud.provider != 'external'
config['cloudProvider'] = @config.cloud.provider
if @config.cloud.config
config['apiServerExtraArgs']['cloud-provider'] = @config.cloud.provider
config['apiServerExtraArgs']['cloud-config'] = CLOUD_CFG_FILE
config['controllerManagerExtraArgs']['cloud-provider'] = @config.cloud.provider
config['controllerManagerExtraArgs']['cloud-config'] = CLOUD_CFG_FILE
end
end
Expand Down Expand Up @@ -117,27 +121,29 @@ def build_extra_sans

# @param config [Pharos::Config]
def configure_internal_etcd(config)
endpoints = @config.etcd_hosts.map { |h|
"https://#{h.peer_address}:2379"
}
config['etcd'] = {
'endpoints' => endpoints
'external' => {
'endpoints' => @config.etcd_hosts.map { |h|
"https://#{h.peer_address}:2379"
},
'certFile' => '/etc/pharos/pki/etcd/client.pem',
'caFile' => '/etc/pharos/pki/ca.pem',
'keyFile' => '/etc/pharos/pki/etcd/client-key.pem'
}
}

config['etcd']['certFile'] = '/etc/pharos/pki/etcd/client.pem'
config['etcd']['caFile'] = '/etc/pharos/pki/ca.pem'
config['etcd']['keyFile'] = '/etc/pharos/pki/etcd/client-key.pem'
end

# @param config [Hash]
def configure_external_etcd(config)
config['etcd'] = {
'endpoints' => @config.etcd.endpoints
'external' => {
'endpoints' => @config.etcd.endpoints
}
}

config['etcd']['certFile'] = '/etc/pharos/etcd/certificate.pem' if @config.etcd.certificate
config['etcd']['caFile'] = '/etc/pharos/etcd/ca-certificate.pem' if @config.etcd.ca_certificate
config['etcd']['keyFile'] = '/etc/pharos/etcd/certificate-key.pem' if @config.etcd.key
config['etcd']['external']['certFile'] = '/etc/pharos/etcd/certificate.pem' if @config.etcd.certificate
config['etcd']['external']['caFile'] = '/etc/pharos/etcd/ca-certificate.pem' if @config.etcd.ca_certificate
config['etcd']['external']['keyFile'] = '/etc/pharos/etcd/certificate-key.pem' if @config.etcd.key
end

# @param config [Hash]
Expand Down Expand Up @@ -189,19 +195,13 @@ def volume_mounts_for_authentication_token_webhook
# @param config [Hash]
def configure_kube_proxy(config)
config['kubeProxy'] = {
'config' => {
'featureGates' => {}
}
'config' => {}
}

if @config.kube_proxy.mode
config['kubeProxy']['config']['mode'] = @config.kube_proxy.mode
end

if @config.kube_proxy.mode == 'ipvs'
config['kubeProxy']['config']['featureGates']['SupportIPVSProxyMode'] = true
end

config
end

Expand Down
86 changes: 44 additions & 42 deletions lib/pharos/phases/configure_dns.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ class ConfigureDNS < Pharos::Phase
title "Configure DNS"

def call
patch_kubedns(
patch_deployment(
'coredns',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this kube-dns -> coredns change cause downtime when upgrading from Pharos 1.2?

Copy link
Contributor Author

@SpComb SpComb Jul 30, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, I didn't test to see how it behaves exactly.

If coredns is the new default, then I don't think we'd want to stick with kube-dns long-term. Options are:

  • stick with just kube-dns for 1.3 and deal with it later
  • mandatory migration to coredns for 1.3
    • avoids any need to implement support for switching between kube-dns/coredns
  • provide a gradual migration path for kube-dns -> coredns:
    • use coredns for new installs
    • keep kube-dns for existing installs
    • provide a cluster.yml knob to switch from kube-dns to coredns (or revert to kube-dns if coredns has some regressions)
    • eventually deprecate and remove kube-dns

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

keep kube-dns for existing installs

This would be non-trivial to implement, because it would require kube API access before UpgradeMaster / ConfigureMaster runs and generates the kubeadm config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this kube-dns -> coredns change cause downtime when upgrading from Pharos 1.2?

Nope, it should be smooth ride (kube-dns deployment is removed after coredns is running).

replicas: @config.dns_replicas,
max_surge: max_surge,
max_unavailable: max_unavailable
Expand Down Expand Up @@ -45,52 +46,53 @@ def max_unavailable

# @param replicas [Integer]
# @param nodes [Integer]
def patch_kubedns(replicas:, max_surge:, max_unavailable:)
logger.info { "Patching kube-dns addon with #{replicas} replicas (max-surge #{max_surge}, max-unavailable #{max_unavailable})..." }
def patch_deployment(name, replicas:, max_surge:, max_unavailable:)
logger.info { "Patching #{name} deployment with #{replicas} replicas (max-surge #{max_surge}, max-unavailable #{max_unavailable})..." }

resource = Pharos::Kube.session(@master.api_address).resource(
apiVersion: 'extensions/v1beta1',
kind: 'Deployment',
metadata: {
namespace: 'kube-system',
name: 'kube-dns'
},
spec: {
replicas: replicas,
strategy: {
type: "RollingUpdate",
rollingUpdate: {
maxSurge: max_surge, # must be zero for a two-node cluster
maxUnavailable: max_unavailable, # must be at least one, even for a single-node cluster
}
},
template: {
spec: {
affinity: {
podAntiAffinity: {
requiredDuringSchedulingIgnoredDuringExecution: [
{
labelSelector: {
matchExpressions: [
{
key: "k8s-app",
operator: "In",
values: [
"kube-dns"
]
}
]
},
topologyKey: "kubernetes.io/hostname"
}
]
}
Pharos::Kube.session(@master.api_address).resource_client('apps/v1').patch_deployment(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Patch should be ok here, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the PATCH matches what kubectl set image does:

$ kubectl -v8 -n kube-system set image deployments/coredns coredns=quay.io/kontena/coredns-amd64:1.1.3
...
I0807 11:11:05.700941    8436 request.go:874] Request Body: {"spec":{"template":{"spec":{"$setElementOrder/containers":[{"name":"coredns"}],"containers":[{"image":"quay.io/kontena/coredns-amd64:1.1.3","name":"coredns"}]}}}}
I0807 11:11:05.700985    8436 round_trippers.go:383] PATCH https://167.99.39.233:6443/apis/extensions/v1beta1/namespaces/kube-system/deployments/coredns
...

Not entirely sure what the $setElementOrder is doing, but it seems unnecessary. Apart from that the container image part is identical.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note that this also changes how dns replicas/maxSurge/etc is sent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, those are simple object-level PATCHes, no array merge needed. Verified that the end result is as intended.

name,
{
spec: {
replicas: replicas,
strategy: {
type: "RollingUpdate",
rollingUpdate: {
maxSurge: max_surge, # must be zero for a two-node cluster
maxUnavailable: max_unavailable, # must be at least one, even for a single-node cluster
}
},
template: {
spec: {
affinity: {
podAntiAffinity: {
requiredDuringSchedulingIgnoredDuringExecution: [
{
labelSelector: {
matchExpressions: [
{
key: "k8s-app",
operator: "In",
values: ['kube-dns']
}
]
},
topologyKey: "kubernetes.io/hostname"
}
]
}
},
containers: [
{
name: 'coredns',
image: "#{@config.image_repository}/coredns-#{@host.cpu_arch.name}:#{Pharos::COREDNS_VERSION}"
}
]
}
}
}
}
},
'kube-system'
)
resource.update
end
end
end
Expand Down
Loading