Skip to content

Conversation

@amotin
Copy link
Member

@amotin amotin commented Nov 8, 2025

BRT_RANGESIZE_TO_NBLOCKS() takes number of ranges as its argument. To get number of blocks we should multiply it by the entry size, not divide by it, as it was due to missing parentheses.

Before #17875 this could cause small memory corruptions for vdevs bigger than 64TB+, depending on kernel's smallest memory allocation unit, but the change made the bug more noticeable.

Fixes #17886

How Has This Been Tested?

Not really.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Quality assurance (non-breaking change which makes the code more robust against bugs)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
  • Documentation (a change to man pages or other documentation)

Checklist:

BRT_RANGESIZE_TO_NBLOCKS() takes number of ranges as its argument.
To get number of blocks we should multiply it by the entry size,
not divide by it, as it was due to missing parentheses.

Before openzfs#17875 this could cause small memory corruptions for vdevs
bigger than 64TB, but the change made the bug more noticeable.

Signed-off-by: Alexander Motin <alexander.motin@TrueNAS.com>
@amotin amotin added the Status: Code Review Needed Ready for review and testing label Nov 8, 2025
@adamdmoss
Copy link
Contributor

That looks right, though I don't feel qualified to give it a review+.
Note that this should be backported to both 2.3 and 2.2 when approved. (@tonyhutter)

@adamdmoss
Copy link
Contributor

Also @robn should be aware though they probably are :)

@alex-moch alex-moch mentioned this pull request Nov 9, 2025
@mabod mabod mentioned this pull request Nov 10, 2025
14 tasks
Copy link
Contributor

@behlendorf behlendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. As you said, it seems this issue has been lurking for a while but hidden by the allocator. It'd be nice to add a KASAN builder to help catch this kind of thing, but I'm not aware of any distro debug kernels which ship with it enabled. We used to build our own kernel for this but it was quite the hassle to maintain.

@behlendorf behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Nov 10, 2025
@behlendorf behlendorf merged commit 8aaed7d into openzfs:master Nov 10, 2025
23 of 25 checks passed
@amotin amotin deleted the brt_nblocks branch November 10, 2025 23:57
@jpds
Copy link

jpds commented Nov 12, 2025

@behlendorf Hey there, sorry to go off-topic here - but could you tell me what you would need for that KASAN integration (and any documentation you have at previous attempts at this?)? I have found: https://docs.kernel.org/dev-tools/kasan.html

I can look into coupling https://nixcademy.com/posts/nixos-integration-tests/ with https://uninsane.org/blog/nixos-kernel-hacking/ to automate the building of kernels + virtual machines for testing for this. There's also https://wiki.nixos.org/wiki/Kernel_Debugging_with_QEMU . But I do not know what requirements you have for this beyond CONFIG_KASAN=y.

tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Nov 12, 2025
BRT_RANGESIZE_TO_NBLOCKS() takes number of ranges as its argument.
To get number of blocks we should multiply it by the entry size,
not divide by it, as it was due to missing parentheses.

Before openzfs#17875 this could cause small memory corruptions for vdevs
bigger than 64TB, but the change made the bug more noticeable.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin@TrueNAS.com>
Closes openzfs#17886
Closes openzfs#17915
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Nov 12, 2025
BRT_RANGESIZE_TO_NBLOCKS() takes number of ranges as its argument.
To get number of blocks we should multiply it by the entry size,
not divide by it, as it was due to missing parentheses.

Before openzfs#17875 this could cause small memory corruptions for vdevs
bigger than 64TB, but the change made the bug more noticeable.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin@TrueNAS.com>
Closes openzfs#17886
Closes openzfs#17915
@behlendorf
Copy link
Contributor

That's very interesting. When we've set this up in the past the biggest issue was definitely automating and maintaining the building of a debug kernel configured with the CONFIG_* flags we need. Our needs are pretty minimal, basically:

  1. Regularly built kernel packages with headers and our CONFIG_* settings.
  2. Distribution provided VM image for use with QEMU.
  3. Ability to install these custom kernel packages, ideally through the package manager.

I'd think NixOS would work fine for this. We'd have to add any distribution specific knowledge to our CI scripts, but they're already setup to be extended so it should be straight forward. We just need to recipe. We're in a much better place on the CI front now that everything is run via GitHub actions.

As for the kernel config, I'd start with enabling these checkers. We'd have to see if performance is tolerable with all of the additional checking, and then tweak as needed.

CONFIG_KASAN=y - Kernel Address Sanitizer, https://docs.kernel.org/dev-tools/kasan.html
CONFIG_KMEMLEAK=y - Kernel Memory Leak Checker, https://www.kernel.org/doc/html/v4.15/dev-tools/kmemleak.html
CONFIG_GCOV_KERNEL=y Kernel Code Coverage, https://www.kernel.org/doc/html/v4.15/dev-tools/gcov.html

Does that sound reasonable?

behlendorf pushed a commit to behlendorf/zfs that referenced this pull request Nov 12, 2025
BRT_RANGESIZE_TO_NBLOCKS() takes number of ranges as its argument.
To get number of blocks we should multiply it by the entry size,
not divide by it, as it was due to missing parentheses.

Before openzfs#17875 this could cause small memory corruptions for vdevs
bigger than 64TB, but the change made the bug more noticeable.

Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Signed-off-by: Alexander Motin <alexander.motin@TrueNAS.com>
Closes openzfs#17886
Closes openzfs#17915
@jpds jpds mentioned this pull request Nov 15, 2025
15 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Status: Accepted Ready to integrate (reviewed, tested)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel oops on 6.12.56 with zfs 6cfc3dba9c

4 participants