unzip cannot create non-utf8 files on APFS

Question

Level 1

10 points

unzip cannot create non-utf8 files on APFS

unzip cannot create files on APFS if filename is not in utf8 encoding:

error: cannot create <FILENAME_WITH_NON-UTF8_ENCODING>

Illegal byte sequence

This is frustrating because unzip has no options to change filename encoding, and the original encoding may not be known anyway. Other programs may have the same problem when trying to create a non-utf8 file.

It would be much better for APFS to create a file with any automatic substitution of illegal characters, than to not create the file at all. Is there any environment setting for this behavior?

HFS+ disk image does not have this issue (it substitutes any invalid byte with the 3-byte string %HH, where HH is the byte in hex). As a temporary solution I unzip into an HFS+ disk image and then copy from there into APFS. Is there a better way?

MacBook Pro (13-inch, Late 2016, 4 TBT3), macOS High Sierra (10.13.2), null

Posted on Dec 23, 2017 10:03 AM

Reply

Answer 1

Dec 23, 2017 11:07 AM in response to George Ef

There is no occurrence of UTF-8 in the zip/unzip man page, and although unzip does have a switch (-a) that will convert the line endings of text files, and EBCDIC encoding to ASCII, there is no mention of extraction UTF-8 encoding. The UnArchiver application in its Advanced tab defaults to automatic encoding detection, but it does not explicitly state that it will perform encoding conversion on extracted text files.

zipinfo -s foo.zip

will generate a ls -l listing of the files in the zip archive. What you are looking for is which are text, and which are binary. Next to the filesize, will be either a tx, or bx indicating a text, or binary file. You may have to run the iconv(1) utility on the text files post extraction to get them to UTF-8 encoding.

To generate a postscript version of the these man pages in the Terminal, and then open/convert it to PDF in Preview:

man -t zip | open -f -a Preview

man -t zipinfo | open -f -a Preview

man -t unzip | open -f -a Preview

man -t iconv | open -f -a Preview

Reply

Answer 2

George Ef Author

Level 1

10 points

Dec 23, 2017 11:17 AM in response to VikingOSX

I only wanted to convert the filenames during extraction, not the file content. I use iconv/recode/uconv as you suggested for the content conversion.

By the way iconv seems to have a bug in normalization conversion between utf8 (NFC) and utf8-mac (NFD), sometimes I need to double convert through utf16, or to use a perl script.

Reply