L2ARC: tag deferred ABDs with device to fix multi-device parallel writes #18158
Conversation
The free-on-write list is a global structure shared across all L2ARC devices. When multiple L2ARC devices write in parallel, each device's write completion must only free its own deferred ABDs to prevent one thread from destroying ABDs belonging to another device's in-flight writes. This commit adds device tagging to the l2arc_data_free_t structure by storing the owning l2arc_dev_t pointer with each deferred ABD. The l2arc_do_free_on_write() function now accepts a device parameter to selectively free only ABDs belonging to that specific device, preventing cross-device interference during parallel L2ARC write operations while maintaining the global list structure for simplicity.
|
Thanks for the work. However, the mentioned patch is already included in #18093, which is currently under review. |
thanks for your feedback |
I don't believe this. The code in OpenZFS writes only one L2ARC device at a time, so I don't know who was testing what. This looks like AI hallucination. |
I can picture this isolated fix being valuable for OpenZFS 2.3.6 and 2.4.x while the full #18093 rework may be considered too invasive for a point-release. Not my call though. |
|
@adamdmoss - Thanks for the feedback. Parallel multi-device write support is introduced in #18093. Before that, a single global feed thread wrote to all devices sequentially, so I don’t believe we could have a regression without per-device feed threads. |
Right! Sorry. amotin's comment makes more sense now. |
4 participants
Motivation and Context
When multiple L2ARC devices perform writes in parallel, the free-on-write list—being a global structure shared across all devices—can cause one device’s write completion to free ABDs belonging to another device that still has in-flight writes. This may result in memory corruption and hard-to-debug race conditions under concurrent workloads.
We developed an internal tool to analyze downstream ZFS forks and identify potentially valuable fixes that have not yet been merged upstream. Using this tool, we discovered the following commit in the TrueNAS ZFS fork:
truenas@69942e1
The change addresses a real concurrency issue in the L2ARC multi-device write path, and we believe it is generally applicable, so we are proposing it upstream for review.
Description
The free-on-write list is a global structure shared across all L2ARC devices. To prevent cross-device interference during parallel writes, this change tags each deferred ABD with its owning
l2arc_dev_t.The
l2arc_do_free_on_write()function is updated to accept a device parameter, allowing it to selectively free only ABDs associated with the specified device. PassingNULLpreserves the existing behavior of freeing all deferred ABDs during shutdown.This approach maintains the simplicity of a global free-on-write list while ensuring correct and safe behavior under concurrent multi-device L2ARC write operations.
Credit goes to the original authors in the TrueNAS ZFS project.
How Has This Been Tested?
This change was validated through code review and targeted stress testing of parallel L2ARC write paths. The updated logic ensures that deferred ABDs are freed only by their owning device, eliminating cross-device races under concurrent write workloads.
Types of changes
Checklist