Split and Compress files

Hello

You may try something like this.

# (2gb) abc.mp4  =>  (1gb) abc_split.z01 + (1gb) abc_split.zip
zip -s 1g abc_split.zip abc.mp4

# no compression by zip, for mp4 is already well-compressed
zip -0 -s 1g abc_split.zip abc.mp4

# unzip split zip archives (specify the last segment)
unzip abc_split.zip

Regards,

H

I use gzip and gunzip, they seem to be more flexable in reading and writing to standard inpiut and output making this all easier.

So to compress and split a file I would do

gzip -c IMG_0092.jpg| split -b640k - outFile

The options are

• -c write the output to standard out
• the file we are compressing
• the pipe symbol - take the output of gzip and send it to the next program
• split is the next program
• -b split the input into this size files, in this case 640 kb (the k says kb, m would indicate mb)
• - take the input from standard in
• outfile, use this as the prefix for the split files

The output files will be named

• outFileaa
• outFileab
• outFileac
• etc

to put them it all back together and recover the original do

cat outFilea? | gunzip > orignalFile.jpg

now orignalFile.jpg is a reconstruction of IMG_0092.jpg

In your case you would need to play around with the split size (1024m should do for what you asked) and of course the names of the files will be different.

There is a program zipsplit that will compress and split files. You can also use the two programs zip (or on of its many spinoffs) and split. Pipe the output of zip into split. This is the way I have always done it, split seems to give more control over the process.

All this depends on what you are looking to do. Do you want to compress and split individual files or combine a bunch of files into one compress archive and spit that?

Finally don't overlook tar, useful if you are looking to archive, compress and split a large directory tree.

there are plenty of compression tools on the mac. gzip and bzip2.

Lately, I like bzip2. This works a little easier on linux, especially if

the original file is split into a lot of smaller ones.

$ gzip TestFile

$ ll

total 120944

-rw-r--r--@ 1 andya 501 61921000 May 31 22:01 TestFile.gz

$ split -b 10m TestFile.gz

$ openssl dgst -sha256 TestFile.gz xa*

SHA256(TestFile.gz)= cd041d79b4af1a54b524602363a18e67201c4acb03675dfebcae9109d8707367

SHA256(xaa)= a3d803049aee16cbbfd679668164707eb9053488fb2ec5720f282a711ee8c451

SHA256(xab)= 0a79e26c77cb47ec09f5cf68cfa45ea8f52f5157cad07c0ac187eaf0ae59ff79

SHA256(xac)= 0f556e8e93dcb41cb3ab20454ab46c016d6596316d75316d810f45e7c2b3682e

SHA256(xad)= abc3db83737346a8af6ac7ba9552c4b71cf45865f7b9faded54f1683b2afd077

SHA256(xae)= 3afbad7b68a1d1c703865422e40cbd68ca512a652f985a0714258b7d936ad0f6

SHA256(xaf)= 11879853fcfbe6df6fb718e1166d4dcae7e0e6ebd92be6c32c104c0a28f0439a

keep the hash of the smaller file, in case you get a error on the far end of the transfer.

That way you only need to resend the small file that's corrupt.

put the the TestFile back together.

$ cat xa* > ScratchFile

$ openssl dgst -sha256 TestFile.gz ScratchFile

SHA256(TestFile.gz)= cd041d79b4af1a54b524602363a18e67201c4acb03675dfebcae9109d8707367

SHA256(ScratchFile)= cd041d79b4af1a54b524602363a18e67201c4acb03675dfebcae9109d8707367

To split the file in to 1GB files.

split -b 1000m abc_compressed.zip

you'll actually get 3 files. 2 - 1GB files, and 1- 5MB file.

-rw-r--r-- 1 andya 501 1048576000 Jun 1 09:59 xaa

-rw-r--r-- 1 andya 501 1048576000 Jun 1 09:59 xab

-rw-r--r-- 1 andya 501 50331648 Jun 1 09:59 xac

This is just something to be aware of.

To actually split it at the command line into 2 - 1GB files run:

split -b 1024m abc_compressed.zip

ll

-rw-r--r-- 1 andya 501 1073741824 Jun 1 10:03 xaa

-rw-r--r-- 1 andya 501 1073741824 Jun 1 10:03 xab

again, to put these files back together, just run

cat xaa xab > filename.zip

Hope this helps.

Andy

Oops. Unzip code was totally wrong. You need to use cat and unzip. Like this.

# unzip split zip archives (abc_split.z01, abc_split.zip)
cat abc_split.z01 abc_split.zip > abc_joined.zip
unzip abc_joined.zip

(unzip yields warnings about the wrong offsets because concatenated archive is broken in this regards)

Sorry for any confusions I may have made.

Regards,

H

Start with this in the Terminal app:

man zip

Frank Caggiano wrote:

There is a program zipsplit that will compress and split files...

In addition to the zipsplit and split commands, see the csplit command.

Bear in mind that a video file is already compressed. Zipping it will not achieve much further compression.