Bug Report: Possible Memory Leak In com.sun.jts.CosTransactions.LogFileHandle #7849
Description
Brief Summary
Investigating some out of memory errors we encountered in our domains running Payara 6.2025.8, inspecting the heap dumps the issue seemed to be that many thousands of instances of com.sun.jts.CosTransactions.LogFileHandle were being retained, which then held byte arrays of significant size. Each LogFileHandle instance had one strong reference, from a lambda being held as the "action" of a jdk.internal.ref.CleanerImpl$PhantomCleanableRef
On code inspection, in LogFileHandle the Cleaner is setup here:
https://github.com/payara/Payara/blob/main/appserver/transaction/jts/src/main/java/com/sun/jts/CosTransactions/LogFileHandle.java#L229
This code is passing a lambda to the cleaner:
public final void registerDestroyEvent() {
CleanerFactory.create().register(this, () -> {
try {
destroy();
} catch(LogException ex) {
// Ignore it
}
});
}The Cleaner javadoc warns against this:
The cleaning action is invoked only after the associated object becomes phantom reachable, so it is important that the object implementing the cleaning action does not hold references to the object. In this example, a static class encapsulates the cleaning state and action. An "inner" class, anonymous or not, must not be used because it implicitly contains a reference to the outer instance, preventing it from becoming phantom reachable.
...
The cleaning action could be a lambda but all too easily will capture the object reference, by referring to fields of the object being cleaned, preventing the object from becoming phantom reachable.
I believe by calling an instance method (destroy()) the lambda is capturing a reference to the LogFileHandle, preventing it from ever being garbage collected.
This seems to have been introduced by #6179 , which used a similar pattern to replace use of finalize() for several other classes as well, so if this is indeed the problem there are likely other classes with similar issues. I have seen in the heap dump similar potentially leaked instances of com.sun.enterprise.loader.ASURLClassLoader and com.sun.enterprise.deployment.deploy.shared.InputJarArchive, although not in large enough numbers to cause a noticeable issue,
Expected Outcome
LogFileHandle instances become phantom reachable, are cleaned, and then garbage collected.
Current Outcome
LogFileHandle instances are retained indefinitely, eventually causing an out of memory error.
Reproducer
None identified as I do not know how to trigger the creation / cleanup of LogFileHandle, but presumably doing whatever would create and destroy one in a loop would reproduce.
The issue has occurred in two domains running different workloads so it isn't clear if any particular application logic is involved or it is just something that accumulates over time as the domain runs.
Operating System
RHEL 8.10
JDK Version
OpenJDK 21.0.6
Payara Distribution
Payara Server Full Profile