Text attachment Line Feed gets converted

Question

Level 1

7 points

Text attachment Line Feed gets converted

Hi all,

ok, I'm having this strange pb which keeps me from using the Mail.app.

Regularly I do have to send csv files for/to windows users. So I use TextWrangler to save the file with the correct Windows line endings, i.e. Carriage Return + Line Feed (have checked with other tools, but same result).

The strange thing I'm experiencing is that the attachment is modified by the mail app and arrives modified at destination (Outlook Express and/or Outlook).

The mail app adds an extra Carriage Return to each end of line.

Before sending the file looks as follows :

abc;def;ghi<CR><LF>
jkl;mno;pqr<CR><LF>
...

After opening the file at destination, it looks as follows

abc;def;ghi<CR>
<CR><LF>
jkl;mno;pqr<CR>
<CR><LF>
...

Ok, now this only happens if the file extension is CSV. Having the same file attached twice to an outgoing mail (once named abc.csv and once abc.dat) only the abc.csv file gets modified; the abc.dat arrives as it should (without any modification).

Also this conversion is made never mind if I check or uncheck the "send windows compatible attachments".

Prior to Snow Leopard (i.e. with Leopard) this did not occur. This does not occur with Thunderbird either, but I'd prefer to use Mac Mail if possible.

Am I the only one having this issue or is there a solution to it ?

I'd appreciate your kind help.

Thank's a lot.

Bye
Lutz

MacBook Unibody 13", Mac OS X (10.6.2)

Posted on Nov 16, 2009 7:32 AM

Reply

Answer 1

awado

Level 2

220 points

Nov 16, 2009 11:33 AM in response to lume96

I would recommend to send those files in zipped/compressed form. Or is that to complicated for the recipients?

Reply

Answer 2

lume96 Author

Level 1

7 points

Nov 16, 2009 12:22 PM in response to awado

Thanks for your input. Obviously I could zip the files or even just change the extension but as you guessed it, that would mean extra work for the recipient.
Also my main concern is about the fact that Mac Mail converts files without asking. I discovered this just after migrating to Snow Leopard and went directly to Thunderbird.
Since then I come back to mail.app after each system update (10.6.1 and now 10.6.2) to see if things came back to normal. No luck though for the time being.

Reply

Answer 3

awado

Level 2

220 points

Nov 16, 2009 12:33 PM in response to lume96

Somehow I think of a preview related thing. Does it still happen, when you chose not to display the previews of attachments? One of the differences between 10.5 and 10.6 is that 10.6 transforms CSVs to tables in Quicklook. Maybe there's a bug and it touches the data inside.

Reply

Answer 4

etresoft

Level 8

46,476 points

Nov 16, 2009 12:41 PM in response to lume96

Are you sure the recipient (or someone in between) isn't modifying the message? I just tried to send myself a couple of CRLF CSV files and they came through with the original line endings.

Reply

Answer 5

lume96 Author

Level 1

7 points

Nov 16, 2009 1:25 PM in response to awado

Hi,
actually I don't think I use the preview. I just attach the files and send the mail. No Quickview used.

As for the attachments modified by the receiver, I did some tests sending the attachments to one of our office PCs on WinXP with Outlook Express and got the same behaviour. Also don't think that it is related to the mail server as it works fine when I send the files using Thunderbird.

I'm however puzzled that there is no info about this "bug" anywhere on the net. I don't see anything special about my setup that could explain this.

Reply

Answer 6

awado

Level 2

220 points

Nov 16, 2009 1:38 PM in response to lume96

My very last idea is the text encoding. Is it set to automatic? If so, does it make a difference, when you chose another encoding? (WIndows Latin 1 etc.)

Reply

Answer 7

etresoft

Level 8

46,476 points

Nov 16, 2009 1:38 PM in response to lume96

lume96 wrote:
As for the attachments modified by the receiver, I did some tests sending the attachments to one of our office PCs on WinXP with Outlook Express and got the same behaviour.

Try sending it from somewhere other than your office to a location other than your office. It is becoming more and more common for intermediaries to perform virus checking on all attachments. More often than not, attachments get scrambled this way. That is almost certainly what is happening. You have to find an environment where no such modifications take place.

I did my tests on with my local machine, using the built-in SMTP and my local POP server. I also tested with my own ISP, which doesn't do any virus checking. The data was fine on both ends.

Also don't think that it is related to the mail server as it works fine when I send the files using Thunderbird.

You don't know how Thunderbird encoded the message. Perhaps it used 8bit or base64 instead of the quoted-printable that Apple uses. That might enable the attachment to get through. However, there is nothing wrong with using quoted-printable. If your mail gateways are changing that file, they are the ones that need to make changes.

I'm however puzzled that there is no info about this "bug" anywhere on the net. I don't see anything special about my setup that could explain this.

The "bug" is specific to your environment.

See if you can take a look at the raw message that Thunderbird sends. Here is what Apple Mail generated for my little test:


--Apple-Mail-4-923865396
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
 charset=us-ascii
plain text
--Apple-Mail-4-923865396
Content-Disposition: attachment;
 filename=test.csv
Content-Type: text/csv;
 x-mac-type=54455854;
 x-mac-creator=21526368;
 x-unix-mode=0644;
 name="test.csv"
Content-Transfer-Encoding: quoted-printable
field1,field2,field3=0D
hello,this,is=0D
a,test,of=0D
csv,file,format=0D
--Apple-Mail-4-923865396--

There are definitely bugs in Apple Mail and other items I think are completely broken. Usually, however, they were broken by Apple on purpose just to make all the even-more-broken PC e-mail clients more likely to read Apple messages. They have been trying to dumb down their e-mail clients for over 10 years in an effort to get PCs to read messages that have always conformed to published e-mail standards.

Reply

Answer 8

lume96 Author

Level 1

7 points

Nov 16, 2009 7:31 PM in response to etresoft

Hi all,

first I'd like to thank everybody for trying to help.

@awado : I've tried other encoding, but it does not make any difference.

@etresoft : Followed your advise and set up two other smtp servers (one of them being the GMail SMTP server) than the one I usually use in the office. Still getting the same issue.

Also, as I can see from your copy-paste of the Mail message, your initial csv file only has the CARRIAGE RETURN line ending (=0D). On my side CSV files with only the <CR> line ending come through nicely as well and are not altered. I only get in trouble when sending a csv file with the Windows-type line ending (i.e. <CR><LF>).
Did some other tests concerning file extension and when sending the same file with and .xml extension instead of .csv, it comes through correctly. Only .csv affected, very strange.

Bye.

Reply

Answer 9

lume96 Author

Level 1

7 points

Nov 17, 2009 1:34 AM in response to lume96

Hi,

ok, I've done some further testing:

1) Took a given file with CARRIAGE RETURN and LINE FEED, i.e. =0D=0A at the end of each line.

2) made two copies of this file (one with .csv extension, the other with .xml extension).

3) Attached both files to a message to myself.

Here's what the raw message of the received mail in my inbox looked like:

_For the CSV file :_

Content-Disposition: attachment;
filename=xyz-virgule.csv
Content-Type: text/csv;
x-unix-mode=0644;
name="xyz-virgule.csv";
Content-Transfer-Encoding: quoted-printable

20091116,1230,EMMBAZ08,,,,,,,,,201600,1=0D
20091116,1230,MILCIT75,,,,,,,,,201414,1=0D
20091116,1230,MILCIT75,,,,,,,,,201522,3=0D

_For the XML file :_

Content-Disposition: attachment;
filename=xyz-virgule.xml
Content-Type: application/xml;
x-unix-mode=0644;
name="xyz-virgule.xml";
Content-Transfer-Encoding: quoted-printable

20091116,1230,EMMBAZ08,,,,,,,,,201600,1=0D=0A=
20091116,1230,MILCIT75,,,,,,,,,201414,1=0D=0A=
20091116,1230,MILCIT75,,,,,,,,,201522,3=0D=0A=

Surprisingly enough, the line ending of the two files is not the same after being encoded to quoted-printable.

So I took a closer look to the quoted-printable reference and found out that the equal sign by itself at the end of each encoded line is a "soft line break". If this "soft line break" is present at the end of an encoded line, NO line break is added to the end of the decoded line. If it is not present, a line break is added to the end of the decoded line.

This explains why I get the <CR><CR><LF> in my CSV on arrival. Each encoded line ends with an =0D (no soft line break present), so a line break is added on decoding, i.e. =0D + =0D=0A (<CR><CR><LF>)

On the other side, each line of my encoded XML file contains a soft line break (=0D=0A =). So on decoding the line, no other line ending is added and I end up with the expected =0D=0A (<CR><LF>).

@etresoft : Now having another look at your last message and the raw message you provided, you're actually suffering the same problem. If you take your encoded lines :

field1,field2,field3=0D
hello,this,is=0D
a,test,of=0D
csv,file,format=0D

and paste them into an online "quoted-printable decoder" you'll get the same altered output I'm getting (a blank line after each data line due to the extra line break) :

field1,field2,field3

hello,this,is

a,test,of

csv,file,format

Ok, so now I know the reasons for the strange output file on arrival but I still do not know why the CSV file is encoded the way it is. Looks like some sort of bug to me in the end.

I was about to set up my laptop from scratch but as etresoft has the same output, I do no longer believe that the problem is specific to my laptop/installation.

Anyone any ideas ?

Thanks for reading to the end 😉

Bye,
Lutz

Reply

Answer 10

etresoft

Level 8

46,476 points

Nov 17, 2009 7:34 AM in response to lume96

lume96 wrote:
ok, I've done some further testing:

Excellent work. Instead of whining, you did some research.

@etresoft : Now having another look at your last message and the raw message you provided, you're actually suffering the same problem.

Yes, you are correct.

Ok, so now I know the reasons for the strange output file on arrival but I still do not know why the CSV file is encoded the way it is. Looks like some sort of bug to me in the end.

That is exactly what it is. I am very disappointed to see Apple screw up something as basic as line endings.

Anyone any ideas ?

I guess you (and I) don't use Mail anymore to send attachments. I don't have any better answer than that. Not only can I not rely on Mail to send attachments, I've been telling people Mail sends attachments just fine. I guess I was wrong. In many cases people may have virus scanners, flaky e-mail clients, and any number of other things that could cause attachments to be corrupted. But now we know that Apple Mail does have bugs that will corrupt your attachments. Thanks Apple. You've made my life more difficult and made me look like a chump. All for line-ending bugs that I and many others learned how to deal with 10 years ago.

I am filing a bug report right away. I'm sorry I didn't notice this sooner. Your research was very good and it was right there in front of me.

Reply

Answer 11

lume96 Author

Level 1

7 points

Nov 17, 2009 8:22 AM in response to etresoft

etresoft wrote:

Excellent work. Instead of whining, you did some research.

Thanks. Actually I hate not understanding something. And given that Google wasn't my friend this time (didn't find anything on that) I tried to get the reasons.

Also I was hoping on some obvious solution as I really like the Apple mail app (except this issue and the fact that you can't insert a table without using an external editor). Also, I haven't found better yet. Thunderbird 3b4 and Postbox are not to bad, but also suffering drawbacks; Entourage, no thanks. What else is there that integrates nicely with the address book and Spotlight ?

etresoft wrote:
I guess you (and I) don't use Mail anymore to send attachments. I don't have any better answer than that. Not only can I not rely on Mail to send attachments, I've been telling people Mail sends attachments just fine. I guess I was wrong. In many cases people may have virus scanners, flaky e-mail clients, and any number of other things that could cause attachments to be corrupted. But now we know that Apple Mail does have bugs that will corrupt your attachments. Thanks Apple. You've made my life more difficult and made me look like a chump. All for line-ending bugs that I and many others learned how to deal with 10 years ago.

I am filing a bug report right away. I'm sorry I didn't notice this sooner. Your research was very good and it was right there in front of me.

I don't know your role other than being a helpful forum member, so I don't really understand why you blame yourself on this.

Could there be a way to force the mail app to identify the csv file as not being of _Content-Type: text/csv;_ ?

Looks like only attachments being identified as text/csv get corrupted whereas other Content-Types are not affected. Although being in the IT for quite some time, I'm NEW to Mac and not familiar with the internals of Mac. However, apparently Content-Type is identified by file extension and not by file - analysis, so maybe there's a way to force the Content-Type in some way by modifying some settings somewhere ?

Bye,
Lutz

Message was edited by: lume96

Reply

Answer 12

etresoft

Level 8

46,476 points

Nov 17, 2009 11:23 AM in response to lume96

lume96 wrote:
What else is there that integrates nicely with the address book and Spotlight ?

Therein lies the problem. I even tried Mail from 10.5 but it refuses to run.

I am filing a bug report right away. I'm sorry I didn't notice this sooner. Your research was very good and it was right there in front of me.

I don't know your role other than being a helpful forum member, so I don't really understand why you blame yourself on this.

I wrote shareware e-mail utility years ago that was big among AOL users who wanted to get their attachments. I blame myself for not noticing it sooner. I even included a snipped of the incorrect attachment, not realizing it was wrong. To be fair to myself, I haven't been neck-deep in e-mail for several years. But I have also been pretty critical with people here claiming that Apple Mail has problems with attachments.

Could there be a way to force the mail app to identify the csv file as not being of _Content-Type: text/csv;_ ?

I know of no way to control how Mail performs its encoding. If you know what your users are doing with the CSV file, you could try convert it to XLS for them perhaps. I tried saving a CSV file from Excel, but Excel saves it with only CR newlines, which Mail preserves, but your PCs wouldn't like.

Looks like only attachments being identified as text/csv get corrupted whereas other Content-Types are not affected

I would have to do a thorough test to make sure of that. Apparently, I would be the first one to run such a test. Linefeeds are a critical part of doing modern e-mail correctly. The fact that Apple has screwed them up now worries me greatly. They had no problem with it in 1995 or 3 months ago.

I'm NEW to Mac and not familiar with the internals of Mac.

It is unfortunate you started with 10.6. You aren't seeing the Mac at its best. Apple Mail (and its predecessors from the early 90s) were rock-solid. 10.6 was billed as a bug-fix and stability release. But then they shipped it early and people have discovered it is, in fact, a major re-architecture of the OS with a little eye-candy.

However, apparently Content-Type is identified by file extension and not by file - analysis, so maybe there's a way to force the Content-Type in some way by modifying some settings somewhere ?

No. I tried a couple of clever ways to get around it. It always screws up the new lines.

Reply

Answer 13

lume96 Author

Level 1

7 points

Nov 17, 2009 11:52 AM in response to etresoft

etresoft wrote:
I know of no way to control how Mail performs its encoding. If you know what your users are doing with the CSV file, you could try convert it to XLS for them perhaps. I tried saving a CSV file from Excel, but Excel saves it with only CR newlines, which Mail preserves, but your PCs wouldn't like.

A bit off-Topic, but actually with Excel you got different types of csv formats in the "Save as" drop down list, one of them being Windows/DOS csv which does save the file with CR LF line endings. That is actually how I do it.

It is unfortunate you started with 10.6. You aren't seeing the Mac at its best. Apple Mail (and its predecessors from the early 90s) were rock-solid. 10.6 was billed as a bug-fix and stability release. But then they shipped it early and people have discovered it is, in fact, a major re-architecture of the OS with a little eye-candy.

I actually started with 10.5.8 and a version of Mail which worked fine. That is part of the pb, I learned to appreciate it 😉

The files are used to be integrated into a Navision IT system of one of our suppliers; they are orders. Therein lies the pb, le person who receives the mail just has to save it to a special location / directory and the rest is done by batch. Therefore, it would be a major drawback to have the person open the file and do a SAVE AS.

I know, they could even automate the whole thing but they prefer keeping it that way.

So I'll have to go back to THunderbird and wait for a bug-fix or another work-around then.

Shame.

Anyway thanks for your kind help.

Bye,
Lutz

Reply

Answer 14

etresoft

Level 8

46,476 points

Nov 17, 2009 12:15 PM in response to lume96

lume96 wrote:
A bit off-Topic, but actually with Excel you got different types of csv formats in the "Save as" drop down list, one of them being Windows/DOS csv which does save the file with CR LF line endings. That is actually how I do it.

I see that. It is a bit funny though. In Excel 2008, there is Windows CSV and DOS CSV. Windows CSV uses CRLF while DOS CSV uses just CR. Plus, Excel does not understand a UTF8 BOM in a CSV file.

The files are used to be integrated into a Navision IT system of one of our suppliers; they are orders. Therein lies the pb, le person who receives the mail just has to save it to a special location / directory and the rest is done by batch. Therefore, it would be a major drawback to have the person open the file and do a SAVE AS.

Actually, I meant for you to save the document differently before saving. However, I cannot come up with a way to send a CRLF CSV using Apple Mail.

I know, they could even automate the whole thing but they prefer keeping it that way.

So I'll have to go back to THunderbird and wait for a bug-fix or another work-around then.

Maybe you could automate it. Instead of AppleMail, you could write a shell script to send the document. An AppleScript wouldn't work because it would want to use Apple Mail. But you can write a short Perl script to compose the mail message and use sendmail to send it.

Of course, using Thunderbird is easier, but automating the process could be beneficial to you.

Reply

Answer 15

lume96 Author

Level 1

7 points

Nov 19, 2009 4:22 AM in response to etresoft

Hi Etresoft,

etresoft wrote:
Maybe you could automate it. Instead of AppleMail, you could write a shell script to send the document. An AppleScript wouldn't work because it would want to use Apple Mail. But you can write a short Perl script to compose the mail message and use sendmail to send it.

I'm actively working on this anyway. In the end, this type of file will be sent/mailed directly by our purchasing software. However there's still some more work to do prior to setting this up.

Today, I'm more concerned about what other files / file-types might suffer wrong encoding.

Let's hope that Apple will correct this bug asap.

Bye,
Lutz

Reply