[Serve] Calls to a Serve Deployment's .remote(), hang after some amount of time / requests.

[JadenFiotto-Kaufman]

What happened + What you expected to happen

After some amount of time and some amount of calls to .remote(), my deployment no longer receives requests. When this happens it corresponds with logs In the Deployment that is calling .remote() like:

"WARNING 2024-09-24 09:51:14,270 serve 20 pow_2_scheduler.py:536 - Failed to get queue length from Replica(id=‘was0z9gr’, deployment=‘ModelDeployment’, app=‘Model:nnsight-models-languagemodel-languagemodel-repo-id-eleutherai-gpt-j-6b’) within 1.0s. If this happens repeatedly it’s likely caused by high network latency in the cluster. You can configure the deadline using the RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S environment variable.”

Here is my setup. I have three services in my architecture.
1.) My own FastApi server.
2.) A Request pre-processing Ray Deployment.
3.) A Compute Ray Deployment.

The FastApi server is my own custom ingress endpoint, not using Ray’s ingress functionality. It connects to an existing Ray cluster via ray.init() on initialization. When the async request endpoint is hit, it uses serve.get_app_handle("RequestApplication").remote(request) to send the request to the Request Deployment.

The Request deployment is on the Ray head node and based on the request received, sends the request to the correct compute deployment like serve.get_app_handle().remote(request). This happens in the deployment’s async call method.

The Compute deployment is on a different ray node on a different machine than the FastApi server and Ray head node. The Compute deployment receives the request also on its async call method, executes the compute heavy request, and handles sending the data back to the FastApi server.

Things to note:

Neither the FastApi server nor the Request Deployment wait for the response from .remote(request) as the Compute deployment handles the result separate from Ray.
The request is a pydantic object which contains a large object as one of its attributes. In the FastApi server, it calls ray.put on that attribute, which then the Compute deployment calls ray.get
In the logs for for the Request deployment, I consistently see “LongPollClient polling timed out. Retrying.” Not sure if this is a problem.
In the FastApi server I sometimes see: “WARNING 2024-09-24 09:51:14,270 serve 20 pow_2_scheduler.py:536 - Failed to get queue length from Replica(id=‘was0z9gr’, deployment=‘ModelDeployment’, app=‘Model:nnsight-models-languagemodel-languagemodel-repo-id-eleutherai-gpt-j-6b’) within 1.0s. If this happens repeatedly it’s likely caused by high network latency in the cluster. You can configure the deadline using the RAY_SERVE_QUEUE_LENGTH_RESPONSE_DEADLINE_S environment variable.”
This is often followed by: “concurrent.futures._base.InvalidStateError: CANCELLED: <Future at 0x7f118c190910 state=cancelled>”
Is there anything I need to do as I’m not waiting for the results of requests to .remote()? Should I be “closing” the DeploymentHandles in some way after I’ve sent data via their .remote() method?

Sorry for the long post. Please let me know if theres any logs or information I can provide.

Versions / Dependencies

Ray: '2.36.0'
Python: '3.10.15'
OS: 'Ubuntu 20.04.6 LTS (Focal Fossa)'

Reproduction script

Not something I can reproduce in one script. Here is a link to our software, specifically the Ray Deployment that calls .remote() and hangs: https://github.com/ndif-team/ndif/blob/dev/ray/deployments/request.py

Issue Severity

High: It blocks me from completing my task.

[JadenFiotto-Kaufman]

Linking discussion: https://discuss.ray.io/t/remote-call-occasionally-hangs/15889/2

[JadenFiotto-Kaufman]

Okay another piece of info, I see this in the python-core-worker log on the non-head node which is hosting the model which often hangs:

[2024-10-07 14:43:24,600 W 460175 460311] core_worker.cc:3310: Failed to report streaming generator return to the caller. The yield'ed ObjectRef may not be usable. object_id=621bcc80a0285281f6d94af4424c32d801c78f380100000003000000

I see it many times.

[JadenFiotto-Kaufman]

@GeneDer Hey would you be available to help debug this issue over a zoom call? I've tried everything I can think of and really don't understand the issue.

[GeneDer]

Hi @JadenFiotto-Kaufman Sorry just had some time to look at this. I have not tested your code locally, but a few things I noted

• If you are trying to serve llm using OSS Ray, you can start with this example https://docs.ray.io/en/latest/serve/tutorials/vllm-example.html
• The way you are using the deployment handle seems like you can probably use Serve's multiplexing feature https://docs.ray.io/en/latest/serve/model-multiplexing.html#model-multiplexing
• You might want to turn on autoscaling on your model deployment https://docs.ray.io/en/latest/serve/autoscaling-guide.html

I will try to see if I can get this deployment to my environment and reproduce the issue and get back to you. But if you haven't already joined Ray's community slack channel, please join via https://www.ray.io/join-slack we can chat more on slack and decide if we need a live debugging session on this :)

[GeneDer]

Hi @JadenFiotto-Kaufman Have you look over the docs I provided and made some changes? I tried to reproduce with the code you share, but it's a bit too complicated for me to run that on my machine. Can you share a simpler example that can just run if you still need help?

[jpambrun]

I think I have the same issue. I wouldn't really be able to share a reproducible example easily, but my deployment performance also degrade over time. I feel like some actors get poisoned/hanged. I have the same "Failed to get queue length from Replica" and "Failed to report streaming generator return to the caller" messages. I am on 2.40. @JadenFiotto-Kaufman, were you ever able to fix it?