How to split and write the output of a linux command to multiple files of a specific size?

Question

How can I split and write the output of my command in Linux bash, to multiple (text) 5M files (number of files does not matter)?

Answer 1

Pipe its output to split.

Example:

$ my-command-name-here | split -a 3 -b 5g - myFile.

will produce files of 5GB having the names myFile.aaa, myFile.aab, myFile.aac a.s.o.

Use -l instead of -b to produce files with the specified number of lines instead of bytes. Read man split or the online documentation.

Answer 2

Another answer that adds the --filter option to support the request from this comment:

my-command | split -b 500m -d --filter='cat > $FILE; zip -m $FILE.zip $FILE' - myFile.

This tells split to use the command provided in parameter --filter to produce the files.

The command dumps the data it receives at stdin into the file $FILE (the variable is set by split with the name of the file it computed) then asks zip to move the file into the archive $FILE.zip.