Hey everyone,
I am currently running into an issue with a GenServer module in my Erlang application that handles background HTTP fetches from external web endpoints. While testing payload retrieval and automated parsing routines, my worker process repeatedly throws an {error, timeout} or drops the connection during socket initialization when attempting to fetch data from a website. I initially assumed the failure was caused by socket pool exhaustion or low default timeout values in :httpc, so I increased the request timeout parameter to 30000ms, enabled keep-alive options, and tested alternative Erlang HTTP client libraries like Hackney, but the process still intermittently fails during socket creation or SSL handshake. Running direct cURL commands outside the BEAM VM works fine, which makes me suspect there might be an issue with how active sockets or connection options are managed when handling responses from this specific host. Has anyone experienced similar socket timeout behavior with regional endpoints in Erlang HTTP clients, or is there a recommended configuration pattern for socket options or transport layers to prevent worker processes from stalling on slow remote servers? Any advice or code snippets would be greatly appreciated.
First thing I notice is no content-length header is being sent in the response when loading that site.
cURL is very tolerant of this, erlang http clients, not so much.
If you can capture the traffic with tcpdump, there’s sometimes pretty revealing patterns of traffic, even if it’s all TLS encrypted.
For problematic servers, disabling keep alive and pipelining (httpc defaults to no pipelining, but just in case you enabled it) can help as well; you may also need to increase max sessions to enable more connections to the back end. Opening a new connection for every request isn’t the most efficient, but it eliminates a lot of potential issues.
would be curious if you reproduce it with latest version of. hackney. If it’s the case please let me know on github 