Skip to content

fix(deps): OTP 27.1.2 → 27.3.4.14 — ssl session-ticket VM crash (#966)#974

Merged
Rasbandit merged 2 commits into
mainfrom
fix/otp-ssl-ticket-crash
Jul 9, 2026
Merged

fix(deps): OTP 27.1.2 → 27.3.4.14 — ssl session-ticket VM crash (#966)#974
Rasbandit merged 2 commits into
mainfrom
fix/otp-ssl-ticket-crash

Conversation

@Rasbandit

Copy link
Copy Markdown
Member

What

Bumps the release/CI toolchain from OTP 27.1.2 to OTP 27.3.4.14 (ssl 11.2.12.10) and hardens rel/env.sh.eex against the undistributed-boot anomaly. Closes #966.

Why: the Jul 3 incident

Prod (OTP 27.1.2) hit repeated GenServer :tls_client_ticket_store terminating (TryClauseError) crashes on TLS 1.3 session tickets from kms.us-east-1.amazonaws.com (bursts 05:02:50 / 05:37 / 06:30 UTC). The store is a singleton under the ssl app supervisor; restart-intensity exhaustion took the permanent ssl app down → VM exit → ~2h of ECS task churn (05:05-07:10 UTC), zero graceful-drain lines in Loki. Full evidence trail in #966.

The fix is in what we ship

OTP 27.3.4.14 / ssl 11.2.12.10 release notes (SSL Release Notes, OTP 27):

A busy client node could self-trigger a ticket store crash if unlucky with scheduling if auto mode is used.
Own Id: OTP-20232 Aux Id: PR-11311

That is exactly our crash class: client-side session_tickets auto-mode ticket store crashing under load. The same release also ships:

Improve error handling of TLS PSK […] Also mend recovery mechanism of ticket and session stores to be as resilient as possible to intermediate bugs.
Own Id: OTP-20216 Aux Id: PR-11282, CVE-2026-55952

so even a future store bug no longer cascades into an ssl-app takedown. Note the OTP-20232 fix landed only in 27.3.4.14 — no earlier 27.x patch carries it — and the bump also picks up the 2026-06/07 ssl CVE fixes (CVE-2026-54887, CVE-2026-54891, CVE-2026-55950, CVE-2026-55952).

Verified the shipped toolchain directly:

$ docker run --rm hexpm/elixir:1.17.3-erlang-27.3.4.14-debian-bookworm-20260623-slim \
    erl -noshell -eval 'application:load(ssl), {ok,V} = application:get_key(ssl, vsn), io:format("ssl ~s~n",[V]), halt().'
ssl 11.2.12.10

Pins changed (Elixir stays 1.17.3)

  • DockerfileOTP_VERSION 27.1.2 → 27.3.4.14, DEBIAN_VERSION bookworm-20241202-slim → bookworm-20260623-slim (the newer erlang tag only exists on the newer snapshot; debian:bookworm-20260623-slim runner tag verified active on Docker Hub)
  • .github/workflows/verify.ymlMIX_BUILDER_IMAGE
  • ci/compose.yml, ci/compose.database.ymlBUILDER_IMAGE
  • .github/setup-runner.shELIXIR_IMAGE
  • docs/context/ci-mix-compile-cache-runner-path.md — present-tense pin reference

Secondary: rel/env.sh.eex undistributed-boot anomaly (#966)

Three tasks in the churn window booted undistributed (hostname -I plausibly returned nothing very early under churn pressure → empty _eni_ipv4 → bogus RELEASE_NODE, node falls back to undistributed). Now: retry up to 5×1s, then fail loudly (exit 1) instead of booting blind to clustering forever — a failed boot gets replaced by ECS; an undistributed boot is invisible.

Notes for reviewers

  • Dialyzer job will cold-build its PLT — the cache key includes the OTP version. Expected one-time slowness, not a failure.
  • Local gauntlet green: mix compile --warnings-as-errors, mix format --check-formatted, mix credo --strict (0 issues), mix sobelow, full mix test.
  • Did not run a full local Docker release build (heavy); the new builder image was pulled and version-verified locally, and CI's prebuild job exercises the full Dockerfile stage.

🤖 Generated with Claude Code

Rasbandit and others added 2 commits July 8, 2026 17:52
Bump the release/CI toolchain to OTP 27.3.4.14 (ssl 11.2.12.10), whose
release notes fix the exact prod crash class from the Jul 3 incident:
tls_client_ticket_store crashing on TLS 1.3 session tickets in auto mode
(OTP-20232, erlang/otp PR-11311), plus ticket/session store recovery
resilience (OTP-20216) and the 2026-06/07 ssl CVE fixes.

Also harden rel/env.sh.eex: retry when hostname -I is transiently empty
during boot, then fail loudly instead of booting undistributed forever
(the #966 secondary anomaly).

Closes #966

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@Rasbandit
Rasbandit enabled auto-merge (squash) July 8, 2026 23:54
@Rasbandit

Copy link
Copy Markdown
Member Author

e2e-clerk + e2e-crdt failures on the first run are known pre-existing flake classes, not the OTP bump: test_34 old-paths-cleaned is open flake #972 (3rd occurrence in 24h, commented there); crdt test_concurrent_edits_both_survive was a one-direction 30s delivery-latency timeout (delivery-flake family — 8/9 crdt + 88/89 clerk passed, stack booted clean on the new image, prebuild green). CI TLS is not in the data path so the ssl bump can't affect delivery timing. Rerunning failed jobs.

@Rasbandit
Rasbandit merged commit 43819ea into main Jul 9, 2026
54 of 57 checks passed
@Rasbandit
Rasbandit deleted the fix/otp-ssl-ticket-crash branch July 9, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

prod: OTP ssl tls_client_ticket_store crash (KMS TLS1.3 session tickets) drove 2h ECS task churn — Jul 3 05:05-07:10 UTC cluster-degraded episode

1 participant