fix(proportion): avoid double-counting inqueue resources for jobs wit…

[Aman-Cool]

What type of PR is this?

bug fix

What this PR does / why we need it:

While investigating some scheduling issues I noticed that jobs can get stuck in Pending even when there's clearly enough queue capacity available. Tracked it down to a double-counting bug in the proportion plugin.

When a gang-scheduled job's tasks get allocated, they move to Binding state in the scheduler cache. The problem is that Binding is part of AllocatedStatus but not ScheduledStatus, so the PodGroup stays Inqueue until the tasks actually reach Running. During this window, the proportion plugin counts those task resources in both attr.allocated and attr.inqueue — the full minResources gets added to attr.inqueue without deducting what's already allocated.

So if job-a uses 3 CPU on a 4 CPU queue, while its tasks are binding the queue looks like it's using 6 CPU, and any new job gets rejected even though there's 1 CPU free.

The PodGroupRunning branch already handles this correctly using GetInqueueResource(job, job.Allocated) which returns max(0, minResources - already_allocated). Applied the same logic to the PodGroupInqueue branch.

Which issue(s) this PR fixes:

Fixes #5099

Special notes for your reviewer:

Added a regression test that runs two scheduler cycles against a shared cache, cycle 1 allocates job-a (3×1CPU) leaving its tasks in Binding, cycle 2 tries to enqueue job-b (1×1CPU) on a 4 CPU queue. Without the fix the test times out waiting for job-b to be bound. With the fix it passes.

Does this PR introduce a user-facing change?

Fix proportion plugin double-counting inqueue resources for jobs with tasks in Binding state, which could cause queues to appear over-capacity and block schedulable jobs.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical bug in the proportion plugin that caused job scheduling issues due to incorrect resource accounting. Previously, jobs with tasks in a Binding state would have their resources double-counted, artificially inflating queue usage and preventing other jobs from being scheduled even when capacity was available. The fix ensures accurate resource allocation by preventing this double-counting, thereby improving scheduler efficiency and preventing jobs from getting stuck in a pending state.

Highlights

• Bug Fix: Resolved a double-counting bug in the proportion plugin where resources for jobs with tasks in Binding state were incorrectly counted in both attr.allocated and attr.inqueue, leading to queues appearing over-capacity and blocking schedulable jobs.
• Resource Calculation Logic Update: Modified the PodGroupInqueue branch within the OnSessionOpen method to deduct already-allocated task resources from minResources when calculating inqueue resources, aligning its logic with that used for PodGroupRunning jobs.
• Regression Test: Added a comprehensive regression test (TestNoDoubleCountingForInqueueJobWithBindingTasks) to validate the fix, simulating a scenario where a job's tasks are in Binding state and verifying that subsequent jobs can be correctly enqueued and bound.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• pkg/scheduler/plugins/proportion/proportion.go
  • Modified the OnSessionOpen method to correctly calculate inqueue resources for jobs with tasks in Binding state.
• pkg/scheduler/plugins/proportion/proportion_test.go
  • Added a new helper function waitForBinds to assist with testing.
  • Introduced TestNoDoubleCountingForInqueueJobWithBindingTasks to validate the fix for resource double-counting.

Activity

• No specific activity (comments, reviews, etc.) has been recorded for this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request provides a solid fix for a double-counting bug within the proportion plugin, which could incorrectly inflate queue resource usage and block schedulable jobs. The change correctly calculates in-queue resources by subtracting already allocated resources for jobs in the Binding state, aligning the logic with how Running jobs are handled. The addition of a comprehensive regression test is excellent, as it clearly demonstrates the bug and verifies the fix, ensuring this issue does not recur.

[Copilot]

Pull request overview

Fixes an accounting bug in the scheduler’s proportion plugin where resources could be double-counted for gang-scheduled jobs whose tasks are in Binding (allocated) while the PodGroup phase remains Inqueue, causing queues to appear over capacity and block otherwise schedulable jobs.

Changes:

• Adjust proportion plugin inqueue accounting to subtract already-allocated resources from minResources when PodGroup is Inqueue.
• Add a regression test that reproduces the Binding-window double-counting and verifies a second job can be enqueued and bound.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
pkg/scheduler/plugins/proportion/proportion.go	Updates PodGroupInqueue inqueue resource calculation to use util.GetInqueueResource(job, job.Allocated) to avoid double-counting.
pkg/scheduler/plugins/proportion/proportion_test.go	Adds a deterministic regression test covering the Binding window and a helper to wait for bind notifications.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Aman-Cool]

Hey @hajnalmt @JesseStutler @hzxuzhonghu, Traced this one down while looking at some scheduling delays; jobs were getting stuck in Pending even with free queue capacity. Turns out the proportion plugin was counting the same resources twice during the binding window. Let me know if you'd like any changes😄.

[hajnalmt]

/retest
/lgtm

Thanks for the contrib! You have awesome findings! Both the problem and the solution is legit. Please port this to the capacity plugin too in an other PR if @JesseStutler agrees. It has the same bug as this one only at two places not just one 😄

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hajnalmt
Once this PR has been reviewed and has the lgtm label, please assign k82cn for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/scheduler/plugins/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Aman-Cool]

@hajnalmt, Thanks so much! Really glad it checks out!😄
I'll take a look at the capacity plugin and open a follow-up PR once @JesseStutler weighs in!

[hajnalmt]

/kind scheduling
/kind bug

👍

[volcano-sh-bot]

@hajnalmt: The label(s) kind/scheduling cannot be applied, because the repository doesn't have them.

Details

In response to this:

/kind scheduling
/kind bug

👍

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[hajnalmt]

I wanted to mention that this shouldn't be here rather than in uthelper, but this is true for getLocalMetrics and getWorkerAffinity too. If you create a new PR for the capacity plugin please move these there too. You will need to have waitForBinds at a common a place anyway.

Also I am thinking that a big chunk of your unit test should be outsourced too. It does its job and I understand that your test does not align with our TestCommonStruct at all, due to the double session openings, but it looks sloppy 😄 Probably we need have a common way for double session tests.

[Aman-Cool]

Makes sense! will keep this in mind while working on the capacity PR😄.
Thanks for bringing this up😁.