I have a situation where I have a Table in Cloudera Impala (Parquet Format),
The table statistcs are:
Size: 23GB Rows: 67M RowSize: Approx 5KB Columns: 308
My Cloudera is Total 6 Nodes Cloudera Cluster (Disk : 84TB Each, Ram: 251GB Each)
Kudu Master and Tablet Server 2 Master Nodes, 5 Tablet Servers (One Node acts as a Tablet Server as well as Master)
here is my table Schema(Structure)
CREATE TABLE SRV_REQ_X
PRIMARY KEY (row_id)
PARTITION BY HASH(row_id) PARTITIONS 5
STORED AS KUDU
TBLPROPERTIES ('kudu.table_name'='IMPALA_DATABASE.KUDU TABLE NAME','kudu.master_addresses'='host1:7051,host2:7051','kudu.num_tablet_replicas' = '3')
AS
Select columns* from table*
Different Properties tested
The properties I have checked and played with are
memory_limit_hard_bytes = Checked with 0 and 1 and 250GB (Same result Tablet
Server Crashes)
maintenance_manager_num = Checked with 1 as well as 4
Records are inserted but at some point this error Comes
Kudu error(s) reported, first error: Timed out: Failed to write batch of 94 ops to tablet 842e935e768f4a419b193e1fb18e3155 after 329 attempt(s): Failed to write to server: 2d35eb2445e747bea574a5e1af6e0b2a (bda-ptcl1node02.ptcl.net.pk:7050): Write RPC to 192.168.228.2:7050 timed out after 179.996s (SENT)
I need to insert other tables which are around 102M records and I cannot understand how to tweak Kudu Properties against my Cluster.
P.S The most records went into the Kudu Table were 13M with Following Properties and then the timeout happened.
memory_limit_hard_bytes = 250GB
maintenance_manager_num = 4
block_cache_capacity_mb = 130GB
Partitions: 4
Please Help!!