Why does sstableloader hang while running it in parallel for multiple keyspaces?
0 S root 8312 8036 24 80 0 - 2304478 futex_ 12:46 ? 00:25:55 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre//bin/java -javaagent:/root/dse-6.8.25/resources/cassandra/bin/../../../resources/cassandra/lib/jamm-0.3.3.jar -javaagent:/root/dse-6.8.25/resources/cassandra/bin/../.............********..................../lib/dse-spark-6.8.25.jar -Dguice_include_stack_traces=OFF -Ddse.system_memory_in_mb=64265 -Dcassandra.config.loader=com.datastax.bdp.config.DseConfigurationLoader -Dclient.hosts=1X.1XX.1XX.1X -Xmx4096M -XX:MaxDirectMemorySize=2G -Dcassandra.storagedir=/root/dse-6.8.25/resources/cassandra/bin/../../../resources/cassandra/data -Dlogback.configurationFile=logback-tools.xml com.datastax.bdp.tools.ShellToolWrapper org.apache.cassandra.tools.BulkLoader -ap com.datastax.driver.dse.auth.DseGSSAPIAuthProvider -cph 2 -d 1X.1XX.1XX.1X /var/lib/cassandra/data/allkey_function22/alldata1-26def030452311ed8bcf47e54b6e7ca3/
Process stack trace :->
pstack 8312 |more
Thread 37 (Thread 0x7f9b10d0d700 (LWP 8313)):
#0 0x00007f9b108f6a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f9b0f790ec4 in Parker::park(bool, long) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre/
lib/amd64/server/libjvm.so
#2 0x00007f9b0f98012e in Unsafe_Park () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre/lib/amd64/ser
ver/libjvm.so
#3 0x00007f9af92ecfea in ?? ()
#4 0x00000006c0014f58 in ?? ()
#5 0x00000006c008d7f8 in ?? ()
#6 0x00000006c0014f58 in ?? ()
#7 0x00007f9b10d0c048 in ?? ()
#8 0x00007f9abc37bfb0 in ?? ()
#9 0x00007f9b10d0c0a0 in ?? ()
#10 0x00007f9abc37c800 in ?? ()
#11 0x0000000000000000 in ?? ()
Thread 36 (Thread 0x7f9b0c8a7700 (LWP 8314)):
#0 0x00007f9b108f6a35 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f9b0f79011b in os::PlatformEvent::park() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre
/lib/amd64/server/libjvm.so
#2 0x00007f9b0f73c327 in Monitor::IWait(Thread*, long) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64
/jre/lib/amd64/server/libjvm.so
#3 0x00007f9b0f73ceaf in Monitor::wait(bool, long, bool) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_
64/jre/lib/amd64/server/libjvm.so
#4 0x00007f9b0f426a9a in GCTaskManager::get_task(unsigned int) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_
9.x86_64/jre/lib/amd64/server/libjvm.so
#5 0x00007f9b0f4285be in GCTaskThread::run() () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre/lib/a
md64/server/libjvm.so
#6 0x00007f9b0f787942 in java_start(Thread*) () from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.345.b01-1.el7_9.x86_64/jre/lib/a
md64/server/libjvm.so
#7 0x00007f9b108f2ea5 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f9b0fff296d in clone () from /lib64/libc.so.6
Thread 35 (Thread 0x7f9b0c7a6700 (LWP 8315)):
seen below exception on destination node :-
e9717] All sessions completed org.apache.cassandra.streaming.StreamState@32eb4373 ERROR [Stream-Deserializer-/10.10.xx.xx:40933-3bd1b85a] 2022-10-15 00:41:24,001 StreamingInboundHandler.java:314 - [Stream channel: 3bd1b85a] stream operation from /10.10.xx.xx:40933 failed java.lang.IllegalStateException: Unknown stream session: 37afd600-4bf3-11ed-9915-d5bc16ce9717 - 1 at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:57) at org.apache.cassandra.streaming.messages.IncomingFileMessage$1.deserialize(IncomingFileMessage.java:48) at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:98) at org.apache.cassandra.streaming.async.StreamingInboundHandler$StreamDeserializingTask.run(StreamingInboundHandler.java:263) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) INFO [OptionalTasks:1] 2022-10-15 00:44:59,282 StatusLogger.java:174 - Pool Name Active Pending Backpressure Delayed Shared Stolen Completed Blocked All Time Blocked AuthzStage 0 0 N/A N/A N/A N/A 0 0 0