PoolTimeout total failure during reuse of httpx.AsyncClient

[kdavh]

We have a server that handles anywhere from 2 - 50 rps. Each server is an AWS r5.2xlarge and has about 40 Gunicorn workers running UvicornWorkers which run a Quart app with lots of custom logic. It's a proxy that uses httpx to make requests to downstreams. We use a single global httpx.AsyncClient to reach out to various downstreams, all controlled by us, some flakey but all serving reasonable responses.
After anywhere from 3-12 hours, one or more worker's httpx.AsyncClient inevitably fails throwing immediate and constant httpx.PoolTimeouts. In the graphs I show below it looks like only one worker fails at a time. These failures don't happen during any particular load period (like no more than 1rps going to downstreams)

We've tried with a bare simple client = httpx.AsyncClient() and much more custom client (custom retrying Transport class, various limits and timeouts configured). They both behave similarly.

Here is a graph showing PoolTimeout being thrown starting around 1:28pm. In normal cases the failure it is rarely thrown, maybe one every few hours:

This is a graph of the pool size (client._transport._pool._pool ) over time. Nothing special seems to happen here.

This is a graph of the request queue size (client._transport._pool._requests ) over time. It looks like it jumps up after the pool timeouts start, but that may be just because a lot of requests come in and are waiting for a connection from the pool before failing.

We've done some debugging but it's hard to iterate because we can't reproduce it locally.

My guesses around possible causes:

• Something around response.read() not being called. My understanding is that httpx calls response.read() before returning the response from await client.request(...) but maybe I'm mistaken. Maybe we should do this just in case on every call (for example some success responses we never need to read the body).
• Some race condition when the server cancels the request for various reasons (like what automatically happens if a client disconnects mid-request) and so a CancelledError is thrown deep inside the httpx stack.
• I would have thought that given that failure only happens after many hours that something like zombie connections slowly builds up, but I don't have any indication that is happening from the graphs I showed. I'm not sure what else to track.

We have successfully mitigated this by retrying requests (for any failure reason) and if a second attempt of client.request(...) ends with httpx.PoolTimeout in particular being thrown, we swap out the global client object for a new one (and we asynchronously call old_client.aclose()). This has run for days with several swaps happening successfully.

Any ideas?

[zanieb]

Hi! Thanks for all the information — not an expert on this one but it may be related to encode/httpcore#550

[kdavh]

That looks very likely, I'll follow progress there.

[Heiaha]

Hi all, I believe this issue is unresolved in the current version (0.28.1). I see the exact behavior described here with AsyncClient, where after approximately one day of requests at 15 r/s, all further requests raise httpx.PoolTimeout. Is this still known to be an issue in the current version?

[fyhertz]

Hi all, we are observing a similar issue with httpx 0.27.2, where all requests fail with httpx.PoolTimeout after a large number of ReadTimeout errors.