Skip to content

[bitnami/postgresql-hq] do not wait for backend nodes in pgpool deployment #1868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sameersbn merged 2 commits into from
Jan 31, 2020
Merged

[bitnami/postgresql-hq] do not wait for backend nodes in pgpool deployment #1868

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3cad537
[bitnami/postgresql-hq] do not wait for backend nodes in pgpool deplo…
Jan 28, 2020
df9db77
[bitnami/postgresql] moved liveness probe command to `healthcheck.sh`…
Jan 31, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bitnami/postgresql-ha/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
apiVersion: v1
name: postgresql-ha
version: 1.4.0
version: 1.4.1
appVersion: 11.6.0
description: Chart for PostgreSQL with HA architecture (using Replication Manager (repmgr) and Pgpool).
keywords:
Expand Down
73 changes: 11 additions & 62 deletions bitnami/postgresql-ha/templates/pgpool/deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,49 +37,6 @@ spec:
{{- $postgresqlHeadlessServiceName := printf "%s-headless" (include "postgresql-ha.postgresql" .) }}
{{- $releaseNamespace := .Release.Namespace }}
{{- $clusterDomain:= .Values.clusterDomain }}
initContainers:

@sameersbn sameersbn Jan 28, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If even a single backend node is unable to come online (think cluster capacity issues) the postgresql-ha deployment will not be ready to accept connections even if we have a working postgresql cluster. We should not have to wait for backend nodes to come online in the pgpool deployment, ideally pgpool would manage the connections. Therefore i've removed this initContainer

- name: wait-for-backend-nodes
image: {{ include "postgresql-ha.volumePermissionsImage" . }}
imagePullPolicy: {{ .Values.volumePermissionsImage.pullPolicy | quote }}
env:
- name: PGPOOL_BACKEND_NODES
value: {{range $e, $i := until $postgresqlReplicaCount }}{{ $i }}:{{ $postgresqlFullname }}-{{ $i }}.{{ $postgresqlHeadlessServiceName }}.{{ $releaseNamespace }}.svc.{{ $clusterDomain }}:5432,{{ end }}
command:
- /bin/bash
- -c
- |
dns_lookup() {
local host="${1:?host is missing}"
getent ahosts "$host" | awk '/STREAM/ {print $1 }'
}
is_hostname_resolved() {
local -r host="${1:?missing value}"
if [[ -n "$(dns_lookup "$host")" ]]; then
true
else
false
fi
}
read -r -a nodes <<< "$(tr ',;' ' ' <<< "${PGPOOL_BACKEND_NODES}")"
declare -i node_counter=0
for node in "${nodes[@]}"; do
read -r -a fields <<< "$(tr ':' ' ' <<< "${node}")"
host="${fields[1]:?field host is needed}"
for ((i = 1 ; i <= 10 ; i+=1 )); do
is_hostname_resolved "$host" && node_counter+=1 && break
sleep 5
done
done
if [[ $node_counter -ne ${#nodes[@]} ]]; then
echo "Not enough active backend nodes!"
exit 1
fi
echo "Every backend node is active!"
exit 0
{{- if .Values.pgpool.securityContext.enabled }}
securityContext:
runAsUser: {{ .Values.pgpool.securityContext.runAsUser }}
{{- end }}
containers:
- name: pgpool
image: {{ include "postgresql-ha.pgpoolImage" . }}
Expand Down Expand Up @@ -171,15 +128,16 @@ spec:
- bash
- -ec
- |
if nodes=$(PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h 127.0.0.1 -tA -c "show pool_nodes;" | grep "up" | wc -l); then
if [[ $nodes -eq {{ $postgresqlReplicaCount }} ]]; then
true
else
false
fi
else
false
fi
if PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h 127.0.0.1 -tA -c "show pool_nodes;"; then

@sameersbn sameersbn Jan 28, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pgpool does not pay nicely with repmgr. if a standby node goes offline pgpool detects it and marks it as down. But when it comes back online pgpool does not update it's status. The only way to make pgpool to update it's status is to restart it. So in the liveness probe we are checking if any of the down nodes are back online and failing the liveness probe if thats the case, thereby triggering a pod restart. Obviously a hacky way of getting things to work consistently.

for node in $(PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h 127.0.0.1 -tA -c "show pool_nodes;" | grep "down" | cut -d'|' -f2); do
if PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h ${node} -tA -c "SELECT 1" >/dev/null; then
false
fi
done
else
false
fi

initialDelaySeconds: {{ .Values.pgpool.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.pgpool.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.pgpool.livenessProbe.timeoutSeconds }}
Expand All @@ -192,16 +150,7 @@ spec:
command:
- bash
- -ec
- |

@sameersbn sameersbn Jan 28, 2020 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the readiness probe simply needs to check if postgresql queries can be executed, we don't need to check if all backend nodes are up

if nodes=$(PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h 127.0.0.1 -tA -c "show pool_nodes;" | wc -l); then
if [[ $nodes -eq {{ $postgresqlReplicaCount }} ]]; then
true
else
false
fi
else
false
fi
- PGPASSWORD=${PGPOOL_POSTGRES_PASSWORD} psql -U {{ (include "postgresql-ha.postgresqlUsername" .) | quote }} {{- if not (empty (include "postgresql-ha.postgresqlDatabase" .)) }} -d {{ (include "postgresql-ha.postgresqlDatabase" .) | quote }}{{- end }} -h 127.0.0.1 -tA -c "SELECT 1" >/dev/null
initialDelaySeconds: {{ .Values.pgpool.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.pgpool.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.pgpool.readinessProbe.timeoutSeconds }}
Expand Down