High RAM consumption by any HttpCrawlers when working with large proxy pools

[Mantisus]

When working with large proxy pools (e.g., Apify RESIDENTIAL), I observe significant RAM usage growth. Memory consumption increases by more than a gigabyte within a few minutes during active scraping.

My version:
The issue appears to be related to creating an HTTP session for each proxy https://github.com/apify/crawlee-python/blob/master/src/crawlee/http_clients/_httpx.py#L132 combined with the high default maximum SessionPool size of 1000 sessions. This results in the crawler creating a new HTTP session for almost every new request during its initial run.

The absence of cleanup logic for created HTTP sessions will likely worsen the situation when the proxy pool contains a large number of "bad" proxies.

[B4nan]

AFAIK this works just fine in the JS version, it feels like a memory leak in the python version somewhere to me. We should investigate why this happens, I don't see why 1000 sessions should be a problem.

Either way, a complete reproduction for this is needed unless you plan to deal with this on your own.

creating an HTTP session for each proxy

There should be a session per proxy. Or better say, there should be a proxy per session, its the other way around, we create sessions, and they are represented by a proxy.

[Mantisus]

I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.

import asyncio
import resource
from httpx import AsyncClient, AsyncHTTPTransport


def log_memory_usage():
    usage = resource.getrusage(resource.RUSAGE_SELF)
    memory_mb = usage.ru_maxrss / 1024  
    print(f"Memory usage: {memory_mb:.2f} MB")

async def create_httpx_sessions():
    sessions = {
        i: AsyncClient(
            http1=True,
            http2=True,
            proxy=f"http://{i}:{i}",
            transport=(AsyncHTTPTransport(
                                    proxy=f"http://{i}:{i}", 
                                    http1=True, 
                                    http2=True))
        ) for i in range(1000)
    }
    return sessions

async def run():
    print("Starting test...")
    log_memory_usage()
    
    sessions = await create_httpx_sessions()

    print("Sessions created")
    log_memory_usage()
    
def main() -> None:
    asyncio.run(run())

Starting test...
Memory usage: 26.57 MB
Sessions created
Memory usage: 1710.93 MB

The same thing happens in the get_client method - https://github.com/apify/crawlee-python/blob/master/src/crawlee/http_clients/_httpx.py#L217

[janbuchar]

I think there is a difference between Python and JS at the level of HTTP clients being used, as memory testing for 1000 httpx.AsyncClient instances shows consumption of about 1.7 gigabytes.

Right, we are supposed to make a separate http client per proxy URL. But... this look like we just make one per request, huh?

[Mantisus]

Right, we are supposed to make a separate http client per proxy URL...

Yes, I understand why we're doing this.

this look like we just make one per request, huh?

Only at the start of the crawler's run. Under ideal conditions (when proxies aren't being blocked), the number of httpx.AsyncClient instances will tend toward the size of the SessionPool. With a large proxy pool, each session will have its own proxy URL. So when the crawler starts working with a SessionPool with max_pool_size of 1000, sessions will rotate with each request, and we create a httpx.AsyncClient for each request, which leads to high memory consumption at the start of crawler run. Once it reaches max_pool_size httpx.AsyncClient instances, the memory consumption stabilizes. The actual memory consumption for 1000 httpx.AsyncClient instances will be higher than in the provided code example since it's not making requests - it will be more than 2 gigabytes. This can be considered the crawler's baseline memory consumption, not counting other loads (Response size, data parsing, queue, caches, etc.)

My main concern is that for an HTTP crawler with default configuration, this memory consumption is very high. For users, this will really look like there's a terrible memory leak somewhere, as Martin suggested.

The situation when working with low-quality proxies will look even worse.

[B4nan]

How costly is creating the HTTP client instance? Maybe we should dispose them the same way as we dispose stale browsers. IMO this is not a good reason to lower the session pool size.

[Mantisus]

About 2 MB per http client instance.

Is large session pool size needed for faster and more efficient scraping?

[B4nan]

I meant how costly it is to create the instance.