Could gen_tcp / ssl expose whether a failed send submitted any data?

[simedw]

I’ve been running into {error, closed} almost daily in production with Erlang/Elixir HTTP clients

The problem is usually the same: a connection has been sitting idle in a pool, the server closes it, and the client tries to reuse it. The request fails, and now we have to decide whether to retry.

I started looking around and found quite a few reports of similar problems across different libraries:

Elixir:

• Finch #272 — Connection closed when request is attempted

• Finch #62 — Mint.TransportError

• Tesla #467 — Proper way to handle closed connections?

• Req #312 — Mint.TransportError - socket closed

• Req #135 — Failing to reconnect after idle

• Mint #279 — Connection errors not being surfaced

• HTTPoison #130 — Reused HTTP connection has been closed

Erlang:

• Hackney #544 — Reused socket closing connection

• Hackney #932 — Connection closed between checkout and request

• Hackney #934 — Keep-alive timer closes a checked-out connection

• ibrowse #147 — Connection selection returns an already-closed connection

• Gun #150 — Connection close and HTTPS

• Gun #346 — Stream closed after HTTP 200

Not all of these are the same bug, and some have already been fixed. But there’s a common problem here that I don’t think can be solved entirely at the connection pool level.

Even if you check that a connection is alive before reusing it, the server can close it a moment later. There’s always a race.

The annoying part is that {error, closed} doesn’t tell you whether anything was actually sent.

For a GET request, retrying is usually fine. For a POST that creates something, not so much. The server might have processed the request before the connection closed, and retrying could create a duplicate.

This has actually happened. In Tesla #467, someone reported getting duplicate requests after adding retries for socket-closed errors.

So I got curious whether OTP itself knows anything more about what happened during a failed send.

I spent some time reading through gen_tcp:send/2 and ssl:send/2, and it turns out OTP sometimes does know that a send failed before anything was submitted. It just doesn’t expose that information.

A few examples:

• In the traditional inet backend, gen_tcp:send/2 can fail during socket lookup, before even calling the backend send.

• prim_inet:send/4 distinguishes between port_command/3 rejecting a command and accepting it but subsequently failing.

In both cases the caller just gets {error, Reason}, even though in the first case we know the send wasn’t submitted.

The newer socket backend is interesting too. gen_tcp_socket:send_result/4 already receives RestData for certain failures, but discards it for non-timeout errors.

TLS has a similar distinction. ssl:send/2 can fail before it even calls the TLS sender, or after data has already gone further down the stack. Same error shape either way.

What surprised me was that some of the information needed for safe retries is already there, but gets thrown away on the way back to the caller. (Unlike for example Rust that gives this back).

So I’m wondering whether it would make sense to expose this distinction in OTP, perhaps with a new function:

-type submission_status() :: not_submitted | unknown.

-spec send_with_status(Socket, Data) ->
    ok | {error, Reason, submission_status()}.

The idea is pretty simple:

• not_submitted: OTP can guarantee that this send failed before anything capable of transmitting its application data was accepted by the underlying transport.

• unknown: OTP can’t make that guarantee.

Importantly, unknown doesn’t mean the request was sent. Just that we don’t know.

And not_submitted says nothing about previous sends on the same connection, or whether the peer received anything from those.

I deliberately avoided a submitted status, since accepting data for transmission doesn’t mean it reached the server either.

This could also be an option on the existing send functions. I’d prefer not to change the behaviour of send/2 itself, since that would obviously break quite a lot of code.

I (and codex) also prototyped this earlier in the week against OTP, then built a small HTTP/1 over TLS test harness to see whether the distinction would actually be useful.

In one run of 500 POST requests:

• 102 failed attempts were classified as not_submitted and safely retried.

• 32 failures were still ambiguous, so I didn’t retry them.

• The server observed zero duplicate executions.

It’s a controlled test, so I wouldn’t draw too many conclusions from the exact numbers. But being able to retry 102 requests that would otherwise have been ambiguous seems quite useful?

Does this seem like something worth exposing at the OTP level and thanks for reading.