The splitted heap dump seems to be messed up. I am not able to combine zip file from it. Anyway visualvm is running with -XX:+HeapDumpOnOutOfMemoryError so in case of OutOfMemoryError the correct heap dump is in C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\ can you please upload this heap dump somewhere (google drive, etc.) and add link here. Thanks.

This issue could be related to #101 though I suspect you have upgraded the Marlin rendering engine that ships with VisualVM at this point.

This is not related to #101, since you are running on JDK 8. FYI: Marlin rendering engine is part of JDK, not VisualVM.

Please see https://youtrack.jetbrains.com/issue/IDEA-305072. Maybe this is one and the same (JDK) bug? This is the application I am attaching VisualVM to.

I don't see how this is related to this bug. Can you please provide the heap dump I asked for? This could give me something I can investigate.

@thurka I've posted an updated heap dump for your review at [removed]. Please let me know when you are done downloading it so I can remove the file. Thank you.

Thanks, I downloaded the heap dump.

@thurka I again think that this issue is related to the JDK hang bug I referenced earlier. If you take a look at https://bugs.openjdk.org/browse/JDK-8296463 Fairoz Matte is only able to reproduce the JDK hang if VisualVM is attached. Something screwy is going on.

Maybe once you fix this bug, the other bug will go away as well. Did you have any luck with the heap dump?

try change
-XX:HeapDumpPath=C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\heapdump.hprof
to 
-XX:HeapDumpPath=C:\Users\Gili\AppData\Roaming\VisualVM\2.1.5\var\log\

I'm not sure,I guess "monitor two JVMs",maybe operate same file or something else, occured an exception:" Could not open PerfMemory", then cause rmi.newSocket again again(maybe some required arguments is missing)--->SEVERE [null]: Last record repeated again.-->OOM

Did you have any luck with the heap dump?

The heap dump shows that the monitored application creates enormenous number of new threads. Threads view needs to store some info for every thread, so this ultimately leads to OOME. The solution is to run VisualVM with increased Xmx. Default is 768M.

@thurka Okay, this is beginning to make more sense. Does VisualVM retain memory for dead threads? If so, increasing the maximum amount of memory won't help because ultimately it'll still run out of memory. As you correctly pointed out, we are creating (and destroying) millions of threads per hour. With JDK 19's ThreadPerTask executor this is going to become a common use-case.

Does VisualVM retain memory for dead threads?

Yes, you can display data for dead threads, so the info for already dead threads are preserved.

If so, increasing the maximum amount of memory won't help because ultimately it'll still run out of memory.

If you plan to monitor it for very long time, yes. With increased Xmx, it can extend the monitoring time long enough, so you can find the information you are looking for (whatever it is).

As you correctly pointed out, we are creating (and destroying) millions of threads per hour. With JDK 19's ThreadPerTask executor this is going to become a common use-case.

Unless I misunderstood JEP 425: Virtual Threads, you should use virtual threads in this situation. You can try it for yourself. In PlatformThreadsTestcase just replace:

    private final ExecutorService blockingTasksExecutor = Executors.newThreadPerTaskExecutor(
            blockingTasksThreadFactory);

    private final ExecutorService blockingTasksExecutor = Executors.newVirtualThreadPerTaskExecutor();

Yes, you can display data for dead threads, so the info for already dead threads are preserved.

This will have to change. It doesn't matter how much memory we throw at VisualVM, the minute an application uses Executors.newThreadPerTaskExecutor VisualVM will run out of memory very quickly. This is increasingly true if anyone uses virtual threads.

Two options that come to mind:

1. Provide a way to disable tracking of Finished threads. If the monitored application uses virtual threads, warn users that they need to disable this feature to avoid OOME.
2. Add a configuration option that reads "Retain history for the last [X] finished threads".

I personally prefer option 2. It would work regardless of the Thread or Executor type being used. It's simple to understand and easy to configure.

Unless I misunderstood JEP 425: Virtual Threads, you should use virtual threads in this situation. You can try it for yourself. In PlatformThreadsTestcase just replace:

I think you misunderstood my point. I meant that once VisualVM adds support for virtual threads (i.e. the Threads tab shows virtual threads, not just carrier threads) then anyone using virtual threads will encounter this OOME problem fairly quickly. We need to fix this problem.

The JDK bug (https://bugs.openjdk.org/browse/JDK-8296463) and IDE bug (https://youtrack.jetbrains.com/issue/IDEA-305777) have both been fixed. The only remaining bug associated with the use of Executors.newThreadPerTaskExecutor is VisualVM.

Have you given any thought on how to solve this? I mentioned two options above. Maybe you have another approach in mind?