Skip to content

system deployments: fix scheduler write skew from stale snapshot#27497

Open
tgross wants to merge 1 commit intomainfrom
NMD1197-systemdeployments-blocked
Open

system deployments: fix scheduler write skew from stale snapshot#27497
tgross wants to merge 1 commit intomainfrom
NMD1197-systemdeployments-blocked

Conversation

@tgross
Copy link
Member

@tgross tgross commented Feb 12, 2026

During a system deployment, if a client update marking an allocation healthy happens after a scheduler worker has taken a snapshot, the plan will overwrite the HealthyAllocs count on the deployment. The client never writes another health update, so the deployment gets stuck even though all allocations are marked healthy.

Likewise, if a scheduler worker takes a snapshot and the deployment is promoted before the plan is submitted, the number of canaries will be mutated and the promotion flag potentially removed.

Break the DeploymentState into "client owned" vs "server owned" fields, and have the update that comes from upserting a plan treat the client owned fields as authoritative for that data. Enforce in the FSM that flipping Promoted to true for the deployment state is a one-way operation.

Fixes: #27382
Ref: https://hashicorp.atlassian.net/browse/NMD-1197

Contributor Checklist

  • Changelog Entry If this PR changes user-facing behavior, please generate and add a
    changelog entry using the make cl command.
  • Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
    ensure regressions will be caught.
  • Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
    and job configuration, please update the Nomad product documentation, which is stored in the
    web-unified-docs repo. Refer to the web-unified-docs contributor guide for docs guidelines.
    Please also consider whether the change requires notes within the upgrade
    guide
    . If you would like help with the docs, tag the nomad-docs team in this PR.

Reviewer Checklist

  • Backport Labels Please add the correct backport labels as described by the internal
    backporting document.
  • Commit Type Ensure the correct merge method is selected which should be "squash and merge"
    in the majority of situations. The main exceptions are long-lived feature branches or merges where
    history should be preserved.
  • Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
    within the public repository.
  • If a change needs to be reverted, we will roll out an update to the code within 7 days.

Changes to Security Controls

Are there any changes to security controls (access controls, encryption, logging) in this pull request? If so, explain.

@tgross tgross added theme/system-scheduler type/bug backport/ent/1.10.x+ent backport to 1.10.x+ent release line backport/1.11.x backport to 1.11.x release line labels Feb 13, 2026
@tgross tgross force-pushed the NMD1197-systemdeployments-blocked branch from 70fdf20 to 6e34f8f Compare February 13, 2026 15:02
@tgross tgross changed the title system deployments: fix race from scheduler having stale snapshot system deployments: fix scheduler write skew from stale snapshot Feb 13, 2026
tgross added a commit that referenced this pull request Feb 13, 2026
This problem isn't especially unique to Nomad, as virtually every web
application in the wild displays write skew. But pointing it out here in our
somewhat unique state store is helpful for onboarding and a good reminder.

Ref: #27497
@tgross tgross marked this pull request as ready for review February 13, 2026 15:20
@tgross tgross requested review from a team as code owners February 13, 2026 15:21
tgross added a commit that referenced this pull request Feb 13, 2026
This problem isn't especially unique to Nomad, as virtually every web
application in the wild displays write skew. But pointing it out here in our
somewhat unique state store is helpful for onboarding and a good reminder.

Ref: #27497
During a system deployment, if a client update marking an allocation healthy
happens after a scheduler worker has taken a snapshot, the plan will overwrite
the `HealthyAllocs` count on the deployment. The client never writes another
health update, so the deployment gets stuck even though all allocations are
marked healthy.

Likewise, if a scheduler worker takes a snapshot and the deployment is promoted
before the plan is submitted, the number of canaries will be mutated and the
promotion flag potentially removed.

Break the `DeploymentState` into "client owned" vs "server owned" fields, and
have the update that comes from upserting a plan treat the client owned fields
as authoritative for that data. Enforce in the FSM that flipping `Promoted` to
true for the deployment state is a one-way operation.

Fixes: #27382
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport/ent/1.10.x+ent backport to 1.10.x+ent release line backport/1.11.x backport to 1.11.x release line theme/system-scheduler type/bug

Projects

None yet

Development

Successfully merging this pull request may close these issues.

system job deployment in 1.11 stalls when all allocations are healthy

2 participants