A socket stuck in FIN_WAIT_2 blocked 800 developers

Google traced weeks of GKE kube-apiserver timeouts to three interacting bugs: NAT port reuse, an Envoy half-close and a Linux kernel FIN bug. What the investigation teaches about debugging shared infrastructure.

A socket stuck in FIN_WAIT_2 blocked 800 developers

dial tcp 10.x.x.x:443: i/o timeout is one of the most boring errors Kubernetes can produce. In one documented case it was the surface symptom of three bugs in three layers — a cloud network fabric, a proxy's connection handling, and the Linux TCP state machine.

Google published the post-mortem on 30 October 2026; InfoQ summarised it on 21 November. Nothing in it is exotic: NAT port reuse, a half-close that never became a full close, and sockets sitting in FIN_WAIT_2 far longer than any timeout promised.

What actually happened

The customer ran a multi-cluster GKE environment in europe-west, with the control plane exposed privately through Private Service Connect. Errors began in January 2026 as intermittent spikes in one UAT cluster, then spread to five. By day 15 a P0 had been filed: roughly 800 developers were blocked from deploying to production.

The obvious suspects were eliminated by measurement, not opinion. A custom kopf operator was cleared because errors predated its deployment by eight days and continued after shutdown. The External Secrets Operator was removed and security agents uninstalled — timeouts stayed. Connection tracking sat at about 50% of a 1.3 million entry default. Control plane VMs were resized; kube-apiserver latency looked healthy and the timeouts did not move.

The symptom was identical at every layer. A timeout on a connection that never completes looks the same whether the fault lives in NAT, in the proxy, or in the kernel.

The three bugs, in order

Packet captures taken simultaneously at the control-plane VM's NIC and inside the kube-apiserver container showed SYNs arriving normally — then being answered by an ACK belonging to a connection that had closed minutes earlier on the same source port. Wireshark labelled those streams "TCP Port numbers reused".

That stale ACK is a challenge ACK, defined in RFC 5961. When a SYN arrives on a 5-tuple the kernel still remembers, TCP replies with an acknowledgement carrying the old sequence number instead of accepting the handshake. A legitimate client answers with a RST, clearing stale state so the next SYN succeeds.

The challenge ACK never reached the client. A Private Service Connect user-space packet switch (USPS) release had been changed to drop packets it classified as invalid ACKs arriving before the handshake completed — exactly what a challenge ACK looks like. They were dropped silently. Private Service Connect NATs many pods behind one endpoint, so 5-tuple collisions happen far more often than on direct connections, and a mechanism the stack had quietly depended on stopped working.

Rolling back the USPS change in February restored connectivity; by early March every affected cluster had recovered. Google is explicit that this was a band-aid. The open question was why the control-plane VM held sockets in FIN_WAIT_2 for over two minutes at all.

Three answers stacked:

  1. Packet reordering at the virtual NIC. Private Service Connect sends the first packet of an unseen flow down a slower path that adds a proxy protocol header and sets up NAT state; later packets take a fast path. A FIN in a closing flow could arrive after the ACKs that followed it, so the kernel saw the close out of order and ignored the client's FIN.
  2. Envoy's half-close froze the timer. tcp_fin_timeout was 60 seconds on those VMs, but it only applies to orphaned sockets. Envoy called shutdown(SHUT_WR) — valid TCP, closing the write side while keeping the socket open to receive. The kernel never treated the socket as orphaned, so the timer never started.
  3. A kernel bug swallowed the FIN. On kernel 6.1, which underpinned the Container-Optimized OS image, tcp_ack() could return zero for an ACK that was old but still inside the RFC 5961 acceptance window. The caller in the FIN path checked that value and did nothing. The FIN arrived and was discarded without error.

The fix was not a flag. It was a Linux kernel patch by Eric Dumazet (commit 795a7dfbc3d9), merged into 6.9.7, backported to the 6.6 and 6.1 LTS branches and delivered to GKE through COS images in 1.33.5 and later. Google also corrected the proxy protocol handling and added a PacketDrill regression test for the challenge ACK path.

What transfers to your own platform

Two habits matter more than the protocol trivia, and they apply to any team running Kubernetes or VPS fleets — including managed infrastructure work at NetPyCloud.

Symptom equivalence is not evidence. Five plausible causes were eliminated, each with a concrete test: log timestamps against a deploy date, the conntrack table read directly, the control plane resized on purpose.

A counter that is flat for months and then spikes is a deployment event, not a symptom. The Proxy protocol drop: invalid ACK packet metric matched a USPS rollout in February, not the week errors began in January. That mismatch was the clue: the drop was introduced later than the symptom, so it could only be an amplifier.

Checklist: sockets that never die

  • Alert on socket age, not count. Sample ss -tan for FIN_WAIT_1, FIN_WAIT_2 and CLOSE_WAIT, and page when one socket survives for minutes.
  • Treat tcp_fin_timeout as covering orphaned sockets only. If a proxy holds the descriptor after shutdown(SHUT_WR), only SO_LINGER and application timeouts bound it.
  • Capture at two points. Reordering between a host NIC and a guest virtual NIC is invisible from either end alone.
  • Correlate anomalous counters with rollout timelines. When the spike date and the incident start date disagree, you are looking at two bugs.
  • Track the kernel and OS image LTS branch, not just the managed service version. This fix depended on a 6.1 backport.
  • Test protocol edges with PacketDrill rather than pcaps. Google's regression for a silently dropped packet runs in CI.
  • Name the interaction in the post-mortem, not a single "root cause". All three behaviours were individually valid, and pinning one would have left the other two in production.

That is the shape of most platform incidents that survive a first review — and why a boring checklist beats a clever one.


Header photo: "AMS-IX optical patch panel" by Fabienne Serriere, CC BY-SA 3.0, via Wikimedia Commons. Sources: Google Cloud Blog, "When the Kubernetes API went silent: A three-layer networking mystery", Akash Thakur, 30 October 2026; InfoQ, Renato Losio, 21 November 2026.