JDK 25 / 0.61.0: SIGSEGV in MM_Scavenger scanning virtual-thread carrier stacks (standalone reproducer)

[nschrader]

Disclaimer: This issue has been AI-investigated. I (human) can confirm that it is real and reproducible as described below.

Summary

We hit intermittent SIGSEGV crashes in the GC scavenger on IBM Semeru / OpenJ9 0.61.0 (JDK 25) in a production service that uses virtual threads. The crashing thread is always a GC worker, and the corrupted reference is always read off a virtual-thread carrier stack.

I have reduced this to a standalone reproducer with no third-party dependencies that segfaults within 25–100 seconds on the stock ibm-semeru-runtimes:open-25-jre-noble image. Two controls isolate the cause:

• the same program on Eclipse Temurin 25 (HotSpot) runs clean
• the same program on OpenJ9 with platform threads instead of virtual threads runs clean

So the trigger is virtual threads on OpenJ9, not the workload itself.

The signatures match two issues that were fixed years ago — #16351 (closed for 0.37) and #21390 (closed for 0.54) — so this looks like a regression or a fresh instance in the same machinery. 0.61.0 is the newest release, so there is no newer build for us to move to.

Environment

openjdk version "25.0.4.1" 2026-08-18 LTS
IBM Semeru Runtime Open Edition 25.0.4.10 (build 25.0.4.1+1-LTS)
Eclipse OpenJ9 VM 25.0.4.10 (build 25.0.4.1+1-openj9-0.61.0, JRE 25 Linux amd64-64-Bit Compressed References 20260818_340 (JIT enabled, AOT enabled)
OpenJ9   - a1daf225f5
OMR      - ebd02d912
JCL      - fd5c7608e81 based on jdk-25.0.4.1+1)

• Image: ibm-semeru-runtimes:open-25-jre-noble (unmodified, no agents, no native libraries)
• Arch: amd64, compressed references
• GC: default (gencon / scavenger)
• Reproduced on Linux 6.8 (local container) and observed in production on Linux 6.12.94 (EKS)
• No JVMTI agent, no profiler, no ASGCT, no JNI code involved — neither in the reproducer (stock image, plain java -jar) nor in the production service (the only -agentlib in our Helm chart is JDWP, disabled by default)

Reproducer

Repro.java is attached below. It has no dependencies. The ingredients were chosen from the production dump:

• virtual threads (carriers appear as ForkJoinPool-1-worker-N)
• direct ByteBuffer allocation, so jdk/internal/ref/PhantomCleanable.<init> becomes hot, gets JIT-compiled, and sits on the carrier stacks — this is the exact method named in the production dump's stack-slot diagnostics
• heavy young-gen churn so scavenges fire constantly while those stacks are deep
• Thread.yield() / LockSupport.parkNanos so continuations mount and unmount mid-frame
• occasional exception stack walks from interpreted frames

# compile
docker run --rm -v "$PWD":/w -w /w \
  ibm-semeru-runtimes:open-25-jdk-noble javac Repro.java

# run (see frequency note below)
docker run --rm --cpus=4 -m 3g -v "$PWD":/w -w /w \
  ibm-semeru-runtimes:open-25-jre-noble \
  java -Xmx1g -Xmns96m -Xmnx96m -XX:MaxDirectMemorySize=1g \
       -Dthreads=2000 -Dseconds=300 Repro

The small, fixed nursery (-Xmns96m -Xmnx96m) just raises the scavenge rate to make the crash arrive sooner; it is not required. Production crashed with default heap settings.

Frequency. The crash is intermittent: 6 of 17 runs (~35%) crashed across 4- and 6-CPU containers with 180–300 s time limits. When it does crash it is fast — observed at 25 s, 40 s, 75 s and 102 s into the run. Expect to run it a handful of times.

Controls

#	JVM	threads	outcome
1	Semeru/OpenJ9 25.0.4.1 (0.61.0)	virtual	SIGSEGV after ~40 s (11.7 M iterations)
2	Semeru/OpenJ9 25.0.4.1 (0.61.0)	virtual	SIGSEGV after ~25 s (4.7 M iterations)
3	Eclipse Temurin 25 (HotSpot)	virtual	clean, 84 M iterations, exited normally
4	Semeru/OpenJ9 25.0.4.1 (0.61.0)	platform (-Dvirtual=false)	clean, 10 M iterations
5	Semeru/OpenJ9 25.0.4.1 (0.61.0), 6 CPUs	virtual	SIGSEGV (4th attempt)
6	Semeru/OpenJ9 25.0.4.1 (0.61.0), 6 CPUs	virtual	SIGSEGV (4th attempt)

Run 4 uses the identical JVM and identical allocation workload and differs only in virtual vs platform threads. Runs 5 and 6 were made with the exact Repro.java attached below, and produced the identical backtrace (calculateObjectDetailsForCopy → copyForVariant → scavengeRememberedSetList, vmState=0x0002000f).

Crash output from the reproducer

Unhandled exception
Type=Segmentation error vmState=0x0002000f
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=000077187EA44A10 Handler2=000077187E62A900 InaccessibleAddress=000000000000001A
RIP=000077187C7B8732 RSP=000077186CD7A3F0
TRAPNO=000000000000000E CR2=000000000000001A
Module=/opt/java/openjdk/lib/default/libj9gc29.so
Module_base_address=000077187C600000
Target=2_90_20260818_340 (Linux 6.8.0-138-generic)
CPU=amd64 (20 logical CPUs)
----------- Stack Backtrace -----------
_ZN22GC_ObjectModelDelegate29calculateObjectDetailsForCopyEP18MM_EnvironmentBaseP18MM_ForwardedHeaderPmS4_S4_+0x32 (libj9gc29.so+0x1b8732)
_ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0x54 (libj9gc29.so+0x191814)
_ZN12MM_Scavenger25scavengeRememberedSetListEP22MM_EnvironmentStandard+0x803 (libj9gc29.so+0x18d5c3)
_ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x2ae (libj9gc29.so+0x18f6be)
_ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0x6f (libj9gc29.so+0x139a5f)
_ZN12MM_Scavenger8scavengeEP18MM_EnvironmentBase+0xcd (libj9gc29.so+0x17967d)
_ZN12MM_Scavenger24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x1c7 (libj9gc29.so+0x1871d7)
_ZN12MM_Scavenger22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0xb21 (libj9gc29.so+0x188741)
---------------------------------------

InaccessibleAddress=0x1A — the scavenger is dereferencing a small integer as an object pointer.

The original production crash (same defect, root-scanning path)

The production instance crashed in the thread root-scanning path rather than the remembered-set path. Both dumps are from the same JVM, moments apart:

Unhandled exception
Type=Segmentation error vmState=0x00000000
Handler1=00007F974BA44A10 Handler2=00007F974B62A900 InaccessibleAddress=0000000000000000
Module=/opt/java/openjdk/lib/default/libj9vm29.so
Target=2_90_20260818_340 (Linux 6.12.94)
CPU=amd64 (8 logical CPUs)
----------- Stack Backtrace -----------
walkStackFrames+0xb03 (libj9vm29.so+0x8d713)
_ZN32VM_BytecodeInterpreterCompressed3runEP10J9VMThread+0xf3e8 (libj9vm29.so+0xd6ee8)
bytecodeLoopCompressed+0xdd (libj9vm29.so+0xc7a6d)
---------------------------------------

followed by stack-slot validator output naming the carrier thread and the JIT'd PhantomCleanable.<init> frame:

0000000001932900: Object neither in heap nor stack-allocated in thread ForkJoinPool-1-worker-8
0000000001932900:	O-Slot=0000000002559F58
0000000001932900:	O-Slot value=00007F96F40374D8
0000000001932900:	PC=00007F96C739F3C1
0000000001932900:	framesWalked=6
0000000001932900:	literals=0000000000000010
0000000001932900:	jitInfo=00007F968FF36B78
0000000001932900:	method=00007F974C3E3F58 (jdk/internal/ref/PhantomCleanable.<init>(Ljava/lang/Object;Ljava/lang/ref/Cleaner;)V) (JIT)
0000000001932900:	stack=00000000025533C8-000000000255A3F0

Note literals=0x10, which is not a valid J9ConstantPool pointer — the interpreter frame linkage is corrupt.

Then the scavenger crashed walking that same stack:

Unhandled exception
Type=Segmentation error vmState=0x0002000f
Handler1=00007F974BA44A60 Handler2=00007F974B62A900 InaccessibleAddress=0000000000000000
Module=/opt/java/openjdk/lib/default/libj9vm29.so
Target=2_90_20260818_340 (Linux 6.12.94)
----------- Stack Backtrace -----------
j9stackmap_StackBitsForPC+0x6d5 (libj9vm29.so+0x1b8e75)
j9cached_StackBitsForPC+0x78 (libj9vm29.so+0x1b7dd8)
walkBytecodeFrameSlots+0x2ff (libj9vm29.so+0x8c8ff)
walkStackFrames+0xbf0 (libj9vm29.so+0x8d800)
_ZN28GC_VMThreadStackSlotIterator9scanSlotsEP10J9VMThreadS1_PvPFvP8J9JavaVMPP8J9ObjectS2_P16J9StackWalkStatePKvEbb+0x39 (libj9gc29.so+0x4a429)
_ZN14MM_RootScanner13scanOneThreadEP18MM_EnvironmentBaseP10J9VMThreadPv+0x117 (libj9gc29.so+0x3eb97)
_ZN14MM_RootScanner11scanThreadsEP18MM_EnvironmentBase+0xc1 (libj9gc29.so+0x3d391)
_ZN14MM_RootScanner9scanRootsEP18MM_EnvironmentBase+0x3e (libj9gc29.so+0x4003e)
_ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x37a (libj9gc29.so+0x18f78a))
_ZN21MM_ParallelDispatcher16workerEntryPointEP18MM_EnvironmentBase+0x19c (libj9gc29.so+0x139ccc)
---------------------------------------

Relation to previously fixed issues

issue	relation	closed in
#21390	same backtrace as our production crash: j9stackmap_StackBitsForPC → walkBytecodeFrameSlots → continuation slot scanning, labels comp:gc/vt	0.54 (Java 24)
#16351	same backtrace as the reproducer: MM_Scavenger::copyForVariant, vmState=0x0002000f, virtual threads under stress	0.37 (Java 19)
#15251	the MM_StackSlotValidator "Object neither in heap nor stack-allocated" diagnostics seen above	0.37 (Java 19)

Possibly related open issue: #23514

#23514 reports a SIGSEGV at the same crash site — walkStackFrames during GC scanSlots under MM_Scavenger, InaccessibleAddress=0x0, with garbage where object pointers should be.

The trigger there looks different from ours: it was diagnosed in-thread as an ASGCT problem (@tajila, 2026-03-17: "This is not a JFR issue, the crash is due to ASGCT"), i.e. asynchronous stack walking from a profiler signal handler racing the GC. We have no agent and no ASGCT involvement at all, and we are on JDK 25 rather than 17.

But both end up with the GC stack walker reading an inconsistent frame, so it is worth considering whether there is a shared underlying fragility in walkBytecodeFrameSlots / the stackmap cache when a frame is observed in a transient state — reached via an async signal walk in #23514, and via continuation mount/unmount here.

Relevant practically: that issue has been stalled since March 2026 because it could not be reproduced — @hangshao0 asked "It will be nice if you have a test program that can reproduce this issue", and @tajila reported being unable to reproduce it. The reproducer attached here is exactly that missing handle on this code path, even if the two triggers turn out to be unrelated.

Secondary problem: the JVM hangs after the GPF instead of terminating

Worth splitting into its own issue, but it is what turned this from a crash into a long outage for us.

After the segfault, the JVM does not exit. It gets stuck here and stays there indefinitely — two crashed containers were left running and were still stuck in this state 9 hours later. In production the pods sat dead until an operator deleted them by hand, 15 and 29 minutes after the crash:

JVMDUMP032I JVM requested JIT dump using '/w/jitdump.20260913.200422.1.0004.dmp' in response to an event
JVMDUMP051I JIT dump occurred in 'ForkJoinPool-1-worker-1' thread 0x0000000000223100
JVMDUMP053I JIT dump is recompiling Repro.lambda$main$1(JI)Ljava/lang/Object;

Because the process never exits, the container is never restarted by Kubernetes: the restart counter stays at 0 and the pod keeps reporting Running while serving nothing. Crash-loop alerting cannot see it.

Disabling only the JIT dump agent is enough to make the process terminate promptly:

flags	trials	crashed	crash → exit	dumps retained
default	2	2	never	all — hangs in the JIT dump agent
-Xdump:jit:none	2	1	6 s	javacore + snap
-Xdump:none	1	1	6 s	none

So the JIT dump agent appears to deadlock or loop when the GPF happened on a virtual-thread carrier and it then tries to recompile the method running there.

What we are doing meanwhile

Using -Xdump:jit:none to bound the damage, and moving the affected code path off virtual threads. Happy to run additional diagnostics — the reproducer is quick, so trace options, debug builds or candidate patches are easy for us to test.

Attachments

Repro.java

[pshipton]

@dmitripivkine pls take a look.

[dmitripivkine]

@nschrader Thank you for reporting this issue, it has been reproduced (with reported stack).
The reason for crash is object in Tenure referenced from Remembered Set (as object pointed to the Nursery) has bad (stale?) pointer:

fragment of RS puddle:

0x7F5D5803D600 :  00000000c01390a0 00000000c0052078 [ ........x ...... ]
0x7F5D5803D610 :  00000000c00e7328 00000000c1a788d0 [ (s.............. ]
0x7F5D5803D620 :  00000000c1277498 00000000c08f4df0 [ .t'......M...... ]
0x7F5D5803D630 :  00000000c0c6b401 0000000000000000 [ ................ ] <--- RS element under scan
0x7F5D5803D640 :  0000000000000000 0000000000000000 [ ................ ]

object in Tenure:

> !j9object 0xC0C6B400
!J9Object 0x00000000C0C6B400 {
	struct J9Class* clazz = !j9class 0xB6500 // jdk/internal/ref/Cleaner
	Object flags = 0x00000010;
	I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
	Ljava/lang/Object; referent = !fj9object 0x0 (offset = 4) (java/lang/ref/Reference)
	Ljava/lang/ref/ReferenceQueue; queue = !fj9object 0xc00e5fe8 (offset = 8) (java/lang/ref/Reference)
	I state = 0x00000001 (offset = 12) (java/lang/ref/Reference)
	I gcLink = 0xC1A77C78 (offset = 16) (java/lang/ref/Reference) <hidden>
	Ljdk/internal/ref/Cleaner; next = !fj9object 0xf6fb41f8 (offset = 20) (jdk/internal/ref/Cleaner) <--- bad pointer to Nursery
	Ljdk/internal/ref/Cleaner; prev = !fj9object 0xc1a77c50 (offset = 24) (jdk/internal/ref/Cleaner)
	Ljava/lang/Runnable; thunk = !fj9object 0xc1b24d18 (offset = 28) (jdk/internal/ref/Cleaner)
}

0xF6FB41E0 :  00000000 00000000 00000000 00000000 [ ................ ]
0xF6FB41F0 :  00000000 00000000 00000000 00000000 [ ................ ] <--- mid object !j9indexableobject 0x00000000F6FB4120
0xF6FB4200 :  00000000 00000000 00000000 00000000 [ ................ ]

There are two observations regarding object !j9object 0xC0C6B400 (may be not related to the problem):

• This is Phantom Reference with referent NULL included to GC list.
• This object happens to be last element in RS puddle.

[dmitripivkine]

@hzongaro Can somebody from JIT team take a look to this problem too while I am investigating GC side of it?
Currently I don't see problem (even potential) at GC side.
If you need existing system core please let me know.
I was able to reproduce the problem on Java 25 (latest nightly build) on xLinux. Reproduction took about 1 hour of test run.
Command line can be simplified to java -Xmx1g -Dthreads=20000 -Dseconds=300 Repro.

[hzongaro]

@mpirvu, is there anyone on the common JIT team that could help investigate to determine whether there could be a JIT problem here?

[dmitripivkine]

An update from another reproduction:

0x7FDB34008DB0 :  00000000c0104910 00000000c1345169 [ .I......iQ4..... ]
0x7FDB34008DC0 :  00000000c133b4d0 00000000c133ced1 [ ..3.......3..... ]
0x7FDB34008DD0 :  00000000c133cd18 00000000c01be0f8 [ ..3............. ]
0x7FDB34008DE0 :  00000000c0904061 00000000c000e590 [ a@.............. ] <--- RS puddle
0x7FDB34008DF0 :  0000000000000000 0000000000000000 [ ................ ]

> !j9object 0xc0904060
!J9Object 0x00000000C0904060 {
	struct J9Class* clazz = !j9class 0xB6500 // jdk/internal/ref/Cleaner
	Object flags = 0x00000010;
	I lockword = 0x00000000 (offset = 0) (java/lang/Object) <hidden>
	Ljava/lang/Object; referent = !fj9object 0xc0904018 (offset = 4) (java/lang/ref/Reference)
	Ljava/lang/ref/ReferenceQueue; queue = !fj9object 0xc0182940 (offset = 8) (java/lang/ref/Reference)
	I state = 0x00000000 (offset = 12) (java/lang/ref/Reference)
	I gcLink = 0xC0904290 (offset = 16) (java/lang/ref/Reference) <hidden>
	Ljdk/internal/ref/Cleaner; next = !fj9object 0xfff989b0 (offset = 20) (jdk/internal/ref/Cleaner) <--- bad pointer
	Ljdk/internal/ref/Cleaner; prev = !fj9object 0xfff76f98 (offset = 24) (jdk/internal/ref/Cleaner)
	Ljava/lang/Runnable; thunk = !fj9object 0xc090a3f8 (offset = 28) (jdk/internal/ref/Cleaner)
}

0xFFF989A0 :  00000000 00000000 00000000 00000000 [ ................ ]
0xFFF989B0 :  00000000 00000000 00000000 00000000 [ ................ ] <--- mid object
0xFFF989C0 :  00000000 00000000 00000000 00000000 [ ................ ]

In this case this is Phantom Reference object as well but referent is not NULL, there is a link to java/nio/DirectByteBuffer object.
And this object is not last (but second to be last) in RS puddle.

[dmitripivkine]

The easiest way to find problematic object (with this crash stack - the same in all reproductions so far)

----------- Stack Backtrace -----------
_ZN22GC_ObjectModelDelegate29calculateObjectDetailsForCopyEP18MM_EnvironmentBaseP18MM_ForwardedHeaderPmS4_S4_+0x32 (0x00007FDB94533CF2 [libj9gc29.so+0x1b3cf2])
_ZN12MM_Scavenger14copyForVariantILb0EEEP8J9ObjectP22MM_EnvironmentStandardP18MM_ForwardedHeader+0x54 (0x00007FDB9450CDD4 [libj9gc29.so+0x18cdd4])
_ZN12MM_Scavenger25scavengeRememberedSetListEP22MM_EnvironmentStandard+0x803 (0x00007FDB94508B83 [libj9gc29.so+0x188b83]) <-----
_ZN12MM_Scavenger24workThreadGarbageCollectEP22MM_EnvironmentStandard+0x2ae (0x00007FDB9450AC7E [libj9gc29.so+0x18ac7e])
_ZN21MM_ParallelDispatcher3runEP18MM_EnvironmentBaseP7MM_Taskm+0x6f (0x00007FDB944B439F [libj9gc29.so+0x13439f])
_ZN12MM_Scavenger8scavengeEP18MM_EnvironmentBase+0xcd (0x00007FDB944F4C3D [libj9gc29.so+0x174c3d])
_ZN12MM_Scavenger24mainThreadGarbageCollectEP18MM_EnvironmentBaseP22MM_AllocateDescriptionbb+0x1c7 (0x00007FDB94502797 [libj9gc29.so+0x182797])
_ZN12MM_Scavenger22internalGarbageCollectEP18MM_EnvironmentBaseP17MM_MemorySubSpaceP22MM_AllocateDescription+0xb21 (0x00007FDB94503D01 [libj9gc29.so+0x183d01])
...

it look to the r14 in the frame for MM_Scavenger::scavengeRememberedSetList()

(gdb) frame 15
#15 0x00007fdb94508b83 in MM_Scavenger::scavengeRememberedSetList(MM_EnvironmentStandard*) ()
   from /team/dmitri/jvm/jdk-25/lib/default/libj9gc29.so
(gdb) i r
rax            0x7fdb90028478      140580990649464
rbx            0x7fdb80245700      140580724430592
rcx            0x7fdb802454b8      140580724430008
rdx            0x7fdb802456a0      140580724430496
rsi            0x7fdb906e3a68      140580997708392
rdi            0x7fdb90063d18      140580990893336
rbp            0x7fdb34008de0      0x7fdb34008de0
rsp            0x7fdb80245630      0x7fdb80245630
r8             0x7fdb802454c0      140580724430016
r9             0x7fdb802454c8      140580724430024
r10            0x7fdb90072240      140580990952000
r11            0x7fdb90063c80      140580990893184
r12            0x7fdb802456c0      140580724430528
r13            0x7fdb9006f610      140580990940688
r14            0xc0904060          3230679136 <--------------------
r15            0x7fdb906e3a68      140580997708392
rip            0x7fdb94508b83      0x7fdb94508b83 <MM_Scavenger::scavengeRememberedSetList(MM_EnvironmentStandard*)+2051>
eflags         0x282               [ SF IF ]
cs             0x33                51
ss             0x2b                43
ds             0x0                 0
es             0x0                 0
fs             0x0                 0
gs             0x0                 0

Now to verify this is the one search for this object low tagged 0xc0904061. This is current object under scan in the Remembered Set puddle.

[mpirvu]

@KavinSatheeskumar Could you please look at this problem from the JIT side? Thanks

[manqingl]

FYI:

I was able to repeat the issue easily (~ 100% chance) in my own Linux machine. After the crash, the process hang there with the following message. I had to kill it with -9.

JVMDUMP053I JIT dump is recompiling Repro.lambda$main$1(JI)Ljava/lang/Object;

The issue can be worked around by adding the following option:

"-Xjit:exclude={Repro.lambda?main?1(JI)Ljava/lang/Object;}"

[x-itm]

The root cause appears to be a stale reference in the Remembered Set during scavenging of virtual-thread carrier stacks. The stack traces consistently show the path through calculateObjectDetailsForCopy → copyForVariant → scavengeRememberedSetList. Since HotSpot and platform-thread variants both pass this test, the issue is specific to OpenJ9's memory management of tenured objects in the garbage collector.

To diagnose further, consider enabling detailed JIT profiling to observe which objects are being retained in the remembered set and their reference counts. Additionally, checking whether the object graph contains circular references or prematurely cleaned-up references could explain the stale pointer. If the reproducer continues to fail after these checks, the fix likely involves adjusting the tenure policy or implementing a custom remembered set filter for virtual-thread carriers.

This type of low-level GC pathology is rare but critical in production. Addressing it requires careful coordination between the JIT compiler and GC subsystem, and may benefit from collaboration with the JIT team to ensure proper reference tracking for long-lived virtual-thread allocations.

[dmitripivkine]

The problem is reproducible with -Xint, just set -Xmn1m and run two test on the same machine simultaneously. This rules out JIT from the picture.
@hzongaro @KavinSatheeskumar FYI