revert(database): drop the PgBouncer use-vc dnsConfig - #4149
Conversation
Reverts e4e3645 (#4135). It was added to force PgBouncer DNS over TCP, but PgBouncer never uses TCP for DNS here. Observed across three pooler pods on two nodes, sampled every 2s for 50s: exactly one connected UDP socket to 10.43.0.10:53 and zero TCP sockets to :53, socket inode unchanged throughout. pgbouncer 1.25.1 resolves through libcares 1.34.5. The option is not inert, though: it applies to every other process in the pod. The pooler's pid 1 is the Go /controller/manager, and Go's net dnsconfig parser honours use-vc, as does glibc NSS. Under the CoreDNS reply drops fixed in #4147 TCP is the worse transport, because a denied SYN,ACK wedges the caller through the full TCP SYN retry ladder while a dropped UDP reply retries in 1-2s. So this removes an aggravating factor, not dead config. Note the mechanism by which c-ares ignores use-vc is not established: c-ares 1.34.5 does parse the option and pgbouncer never passes ARES_OPT_FLAGS to override it. The socket observation is the evidence, not a source-level explanation.
|
Warning Review limit reached
Next review available in: 15 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
@@ spec.template.spec @@
# postgresql.cnpg.io/v1/Pooler/database/pgbouncer-rw
! - one map entry removed:
- dnsConfig:
- options:
- - name: use-vc
@@ spec.template.spec @@
# postgresql.cnpg.io/v1/Pooler/database/pgbouncer-ro
! - one map entry removed:
- dnsConfig:
- options:
- - name: use-vc
@@ spec.template.spec @@
# postgresql.cnpg.io/v1/Pooler/database/pgbouncer-session
! - one map entry removed:
- dnsConfig:
- options:
- - name: use-vc
|
1 participant
Reverts e4e3645 (#4135). Split out of #4147 so the urgent policy fix can merge without a pooler rollout.
Why it never worked
#4135 forced PgBouncer DNS over TCP. PgBouncer never uses TCP for DNS here. Observed across three pooler pods on two nodes, sampled every 2s for 50s:
pgbouncer 1.25.1 resolves through libcares 1.34.5.
The mechanism is not established, and the earlier "c-ares not glibc, so it never takes effect" explanation is wrong: c-ares 1.34.5 does parse
use-vc(ares_sysconfig_files.c: process_option()) and applies it without an optmask gate, and pgbouncer never passesARES_OPT_FLAGSto override it. The socket observation is the evidence; I can't source the why.Why removing it is not cosmetic
use-vcis a pod-leveldnsConfig— it applies to every process in the pod, not just pgbouncer. The pooler's pid 1 is the Go/controller/manager, and Go'snetdnsconfig parser honoursuse-vc/usevc/tcp, as does glibc NSS.Under the CoreDNS reply drops fixed in #4147, TCP is the strictly worse transport: a denied
SYN,ACKwedges the caller through the full TCP SYN retry ladder, where a dropped UDP reply just retries in 1-2s. So this removes an aggravating factor.Blocked on — do not merge yet
This changes
template.spec, so CNPG rolls all 7 pooler pods (rw ×3, ro ×2, session ×2). Two gates:maxSurge=1, maxUnavailable=0.pgbouncer-rwrequests 128Mi; control-3 currently has ~52Mi free (99% of allocatable requested). Onlypgbouncer-rwcarrieswhenUnsatisfiable: DoNotSchedule, so once the topology hits 2/1/0 themaxSkew: 1constraint forces the next surge pod onto control-3 specifically, where it cannot schedule — stalling the roll indefinitely. That is the same deadlock that cost 20 minutes on 07-21, and headroom is worse now than it was then.#4144 frees 2Gi/node by cutting
vm.nr_hugepages2048→1024; landing that first clears gate 2. Alternatively batch this with some future pooler change that has to roll anyway.pgbouncer-roandpgbouncer-sessionuseScheduleAnywayand carry no deadlock risk.