Loading CSV using Mavenir Cassandra loader, getting HectorException: "All host pools marked down. Retry burden pushed out to client."

Question

i am using RHEl 8.6 and Apache-cassandra-4.1.3. Created the cluster with single node and my node is running UP & Normal. as well as I am able to connect cqlsh to query also but When i am trying to upload the a CSV file(data file) from POC staging server side with cassandra-loader to our POC cluster(1node). facing below Error. can you please suggest your valuable idea. Note- Using the command

/mavenir/cassandra/bin/cassandra-loader \
  -l /mavenir/cassandra/cass.log \
  --stats /mavenir/cassandra/cass.log.stats \
  /mis_data/dat/MVNR/sms/load_cass/mo_bill_mhrcn-smsc-02_20231121073956Z_CSV_SM_mo_bill.csv

Directory name is /mavenir/cassandra/bin
Library path is /mavenir/cassandra/lib
Configuration PAth is /mavenir/cassandra/etc
Writing to log file /mavenir/cassandra/cass.log \
  /mavenir/cassandra/cass.log /mavenir/cassandra/etc \
  SLF4J: Class path contains multiple SLF4J bindings. \
  SLF4J: Found binding in [jar:file:/mavenir/cassandra/lib/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] \
  SLF4J: Found binding in [jar:file:/mavenir/cassandra/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] \
  SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. \
  SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 
ERROR StatusLogger Reconfiguration failed: No configuration found for '1be6f5c3' at 'null' in 'null' logger \
  sujathacom.acunu.loader.CassandraLoader logger sujathainfofalse logger \
  sujathacom.acunu.loader.CassandraLoader Sujatha check port9160 \
  Sujatha check serverList10.173.225.159:9042 Sujatha check keyspaceNamehuawei \
  Sujatha check clusterMISMACS_POC
10:54:15.485 [main] ERROR me.prettyprint.cassandra.connection.HConnectionManager - \
  MARK HOST AS DOWN TRIGGERED for host 10.173.225.159(10.173.225.159):9042
10:54:15.487 [main] ERROR me.prettyprint.cassandra.connection.HConnectionManager - \
  Pool state on shutdown: <ConcurrentCassandraClientPoolByHost>: \
  {10.173.225.159(10.173.225.159):9042}; IsActive?: true; Active: 1; Blocked: 0; Idle: 2; NumBeforeExhausted: 9
10:54:15.488 [main] ERROR com.acunu.loader.FileLoader - Fatal exception in thread Thread[main,5,main] \
me.prettyprint.hector.api.exceptions.HectorException: [10.173.225.159(10.173.225.159):9042] \
  All host pools marked down. Retry burden pushed out to client. \
    at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:401) ~[hector-core-2.0-0.jar:?] \
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:232) ~[hector-core-2.0-0.jar:?] \
    at me.prettyprint.cassandra.service.AbstractCluster.describeKeyspaces(AbstractCluster.java:136) ~[hector-core-2.0-0.jar:?] \
    at com.acunu.loader.CassandraLoader.connect(CassandraLoader.java:503) ~[mismacs_mavenir_poc_13_01_24.jar:?] \
    at com.acunu.loader.CassandraLoader.initialise(CassandraLoader.java:432) ~[mismacs_mavenir_poc_13_01_24.jar:?] \
    at com.acunu.loader.FileLoader.loadFiles(FileLoader.java:65) ~[mismacs_mavenir_poc_13_01_24.jar:?] \
    at com.acunu.loader.CassandraLoader.main(CassandraLoader.java:562) ~[mismacs_mavenir_poc_13_01_24.jar:?]
10:54:15.494 [Hector.me.prettyprint.cassandra.connection.CassandraHostRetryService-1] \
ERROR me.prettyprint.cassandra.connection.CassandraHostRetryService - \
  Downed Host retry failed attempt to verify CassandraHost

i installed cassandra in POC server and kept all the jar file of installed cassandra inside the staging server under /mavenir/cassandra/lib/ and created a jar file where all the details like cluster name, destination IP, port number, keyspace name, table name...and tried to run the below command.

/mavenir/cassandra/bin/cassandra-loader \
  -l  /mavenir/cassandra/cass.log \
  --stats /mavenir/cassandra/cass.log.stats \
  /mis_data/dat/MVNR/sms/load_cass/mo_bill_mhrcn-smsc-02_20231121073956Z_CSV_SM_mo_bill.csv

Answer 1

I'm not familiar with the loader tool/utility you are running but it appears it is using the Hector client based on this exception message:

10:54:15.488 [main] ERROR com.acunu.loader.FileLoader - Fatal exception in thread Thread[main,5,main] \
me.prettyprint.hector.api.exceptions.HectorException: [10.173.225.159(10.173.225.159):9042] \
  All host pools marked down. Retry burden pushed out to client.

If your loader utility is using the Hector driver, it won't be able to connect to your cluster because it is using the legacy Thrift protocol.

Thrift was deprecated in Cassandra about 10 years ago, replaced by the CQL native binary protocol all the way back to 2011 (CASSANDRA-1703). Drivers based on the Thrift API were retired including Nate McCall's Hector client in 2015 and Netflix's Astyanax client in 2016.

Cassandra 2.2 stopped using Thrift in 2015 (CASSANDRA-8358, CASSANDRA-9319) and Cassandra 4.0 completely removed Thrift in 2016 (CASSANDRA-11115).

I would suggest using the DataStax Bulk Loader (DSBulk) tool which allows you to bulk load data in CSV or JSON format to a Cassandra cluster. It is fully open-source so it's free to use.

Here are some references with examples to help you get started quickly:

• Blog - DSBulk Intro + Loading data
• Blog - More DSBulk Loading examples
• Blog - Counting records with DSBulk (useful for verifying number of rows loaded)
• Docs - Loading data examples

Cheers!