why chunk files are split even though the file size is not 64MB?

62 Views Asked by At

I was downloading twitter data using flume into hdfs. Though i have data more than 2 GB, my chunk file splits are less than 64MB. i.e. 1st file with 300KB,2nd File - 566 KB. Why is that happening?

1

There are 1 best solutions below

0
On

That is because of your flume configuration.

read this, you will have to set hdfs.rollInterval or hdfs.rollSize