Skip to content

[bugfix] Introduce optional file locking for appending to perflogs#3523

Merged
vkarak merged 5 commits intoreframe-hpc:masterfrom
vkarak:bugfix/lock-filelog-append
Aug 12, 2025
Merged

[bugfix] Introduce optional file locking for appending to perflogs#3523
vkarak merged 5 commits intoreframe-hpc:masterfrom
vkarak:bugfix/lock-filelog-append

Conversation

@vkarak
Copy link
Contributor

@vkarak vkarak commented Jul 17, 2025

This PR introduces two new configuration variables (locking_enable and locking_file_mode) for the filelog handler. Setting this will cause the filelog handler to use file locking when appending to the perflog files.

This PR also introduces the following improvements:

  • Introduces a new utility function osext.flock() to handle the creation of the lock file for the different Python versions; this function is now used also for the DB.
  • Make sure the logging mechanism is shut down correctly when switching configurations and when the reframe exits.
  • Use Python syntax for true and false in the configuration reference to be consistent with the use of None instead of the JSON's null.

Closes #3521.

@vkarak vkarak added this to the ReFrame 4.8.3 milestone Jul 17, 2025
@vkarak vkarak requested review from ekouts and teojgo July 17, 2025 13:16
@vkarak vkarak self-assigned this Jul 17, 2025
@vkarak vkarak changed the title Introduce optional file locking for appending to perflogs [bugfix] Introduce optional file locking for appending to perflogs Jul 17, 2025
@vkarak vkarak force-pushed the bugfix/lock-filelog-append branch 2 times, most recently from a10625f to 353c6d8 Compare July 21, 2025 14:30
Also:

- Introduce an `osext.flock()` utility function to handle the underlying
  locking for the different Python versions.
- Use Python syntax for ``true`` and ``false`` in the configuration
  reference to be consistent with the use of ``None`` instead of the
  JSON's ``null``.
@vkarak vkarak force-pushed the bugfix/lock-filelog-append branch from 353c6d8 to fac5691 Compare July 22, 2025 07:11
@vkarak vkarak marked this pull request as draft July 23, 2025 14:16
@vkarak
Copy link
Contributor Author

vkarak commented Jul 23, 2025

I'm converting this to draft since the fix has to be designed in accordance with #3532.

@vkarak vkarak marked this pull request as ready for review July 24, 2025 10:08
@github-project-automation github-project-automation bot moved this from Todo to In Progress in ReFrame Backlog Aug 11, 2025
@vkarak vkarak enabled auto-merge August 11, 2025 14:08
@vkarak vkarak merged commit 267f8b3 into reframe-hpc:master Aug 12, 2025
37 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in ReFrame Backlog Aug 12, 2025
@vkarak vkarak deleted the bugfix/lock-filelog-append branch August 12, 2025 18:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Perflogs on NFS get corrupted when updated by multiple reframe processes

2 participants