Apache Kudu TServer goes down when I use CTAS (Create Table As) hence my insertion fails

375 Views Asked by At

I have a situation where I have a Table in Cloudera Impala (Parquet Format),

The table statistcs are:

Size: 23GB Rows: 67M RowSize: Approx 5KB Columns: 308

My Cloudera is Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)

Kudu Master and Tablet Server 2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)

here is my table Schema(Structure)

CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU 
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS 
Select columns* from table*

Different Properties tested

The properties I have checked and played with are

memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet 
Server Crashes)

maintenance_manager_num = Checked with 1 as well as 4

Records are inserted but at some point this error Comes

Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)

I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.

P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.

memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB    
Partitions: 4 

Please Help!!

0

There are 0 best solutions below