User profile for user: pterobyte
pterobyte Author
User level: Level 6
11,104 points

To SPF or not to SPF

Since many people keep asking about SPF, I thought I'd publish a few facts gathered from my own experience on servers managed by myself. These are just facts and some personal thoughts. I will not get into any "evangelistic" discussions about SPF.

Although to me it was clear from the beginning that SPF is not a solution I thought I'd give it a try to gather some real world data. In order to be able to analyse the information better, I used a server that has a decent amount of load, but is not exaggeratedly busy.

Basic components used:
Mac OS X 10.4.8 with latest security updates.
Postfix 2.1.5
amavisd 2.4.4
SpamAssassin 3.1.7
ClamAV 0.88.7 with clamd instead of clamscan

SPF related components used:
Mail::SPF and Mail::SPF::Query (tried both in seperate moments)
postfix-policyd-spf-perl (a perl script that calls above PERL script(s) and is used together with postfix' policy server)

I modified postfix-policyd-spf-perl so that it only logs it's "findings", but never actually blocks mail.


Configuration:
I lowered all SPF related SpamAssassin scores to 0.001, because I just wanted to see what gets triggered, but was not interested in action being taken.

SpamAssassin has a well trained Bayes database, all necessary fixes and some additional rules (the same rules that get installed by "spamtrainer -a").

Postfix was configured to reject as much spam as possible before even calling a content filter. (Based on what I published in my document "Frontline spam defense for Mac OS X Server")

The policy server is called after all other checks are done (this is very important, otherwise you can become an open relay if a spammer has actually published an SPF record). It is also more efficient on resources.

I configured the script to log/act on following conditions:
No SPF record = none (equals pass, because rejecting mail from domains without SPF records, basically means ludicrous amounts of false positives)

Correct SPF record and mail sender matches = pass

Correct SPF record but mail sender doesn't match = fail

DNS Errors while looking up = softfail (pass)


So much for the basic information, which is much longer than the statistical data gathered, but nonetheless important for a full picture.

Now to the data:
Test Period: 2 weeks
Average number of incoming mails per day: 32'000
Average number of rejected mails per day: 28'000
(Yes almost 90% of mail is spam or viruses)

Now the juicy bit. Out of the average of 28'000 mails caught by Postfix, ClamAV and SpamAssassin, how many do you think would NOT have been caught without SPF checking in place?:
100 or roughly 0.35%

I got "SPF fail" on an average of 208 mails. 108 of them had been caught by the other measures in place already.

So is this worth the effort and the extra hit on server and network resources? Hardly. At least not in my book.


Remarks:

Why is SPF praised, yet at the same time so inefficent?

The main problem is that it relies on domain owners publishing a voluntary SPF TXT DNS record. This record declares which servers may or may not send mail for said domain. It can be defined as a "strict" record (listing allowed machines and disallowing any other server) or as a "loose" record (listing allowed machines, but not excluding that other servers may send mail for the domain). Most published SPF records are not strict (google.com for example). Why? Because a domain's user on the move forced to use a different ISP's mail server would not be able to send mail anymore (yes I know you can always have different submission ports to prevent being blocked from accessing your own server, but not everybody does). Now when an SPF record is not "strict", rejecting mail not coming from designated servers, yields a very high risk of false positives and thus, most sysadmins will let it through (maybe with an increased SpamAssassin score, but that's about it).

Another problem is that an enormous amount of domains do not publish an SPF record at all (mac.com for example doesn't). Do you want to block those mails? Probably not.


Should I use SPF checking?

This is something you will have to decide for yourself. I am definitely not using it. On my servers it has proven to be an ineffective method of catching spam. It eats cpu cycles and does network roundtrips without a real benefit. It's not that the perl scripts and dns lookups bring my system to its knees, far from it, but I have a strict policy on wasting resources.


Should I publish an SPF record for my domain(s)?

Again, you will have to decide for yourself. Under normal circumstances a mail server will let your mail through even if you do not publish an SPF record. There are however some inept system administrators that do block mail from domains without an SPF record). In many cases, not having an SPF record is safer than having a strict one. I have decided to publish an SPF record for my domains. This doesn't mean I believe in it. It's just something that doesn't cost me any extra resources. Plus I have full control over my servers and do not rely on any ISP doing configuration work for me.


I didn't get into the technical tidbits about SPF's reliabilty, spoofing and so on. There is enough information on the internet on this. All I wanted was to illustrate a real world situation.

Feel free to post questions.

Alex

Mac OS X (10.4.8)

Posted on Jan 2, 2007 3:50 AM

Reply
20 replies
User profile for user: tobias Eichner
tobias Eichner
User level: Level 3
574 points

Jan 2, 2007 4:50 AM in response to pterobyte

Thanks for this thread and sharing your real-world-experiences with SPF... now I need not to borrow an other thread 😉

You asked for personal opinions/questions, so here are mine:

Spam is a huge problem, surely not just for both of us... for example, I get around (at least roughly estimated) 500 to 800 spam mails daily in the meantime; around one year before it was a maximum of 400 on some days, averagely 300. So it grows even more.

Therefore any action I can take to prevent one single spam message being delivered is worth to spend some time and get realized. So I also did with the SPF checks enabled at SpamAssassin.

Unfortunately, so far it hasn't helped much - seen the fact that the amount of spam that comes through was not reduced. But keeping an eye on the CPU and network usage, I saw no impact; neither on my nor on our clients' server. Maybe all happened is that mails already marked as spam are rated a bit higher (would need to verify this statement, since we simply delete all spam mails directly on the server).

But back to topic regarding SPF being useful or not:

Also here I simply can say, it is a nice approach of making things safer, but surely not "the" solution. We use a SPF record in all of our domains, so should there be a recipient mail server persisting upon getting one, here it is. Otherwise I don't care much.

The SPF record we are using: "v=spf1 a mx ~all"

So you see it is simply the default ("softfail") one with less to no impact. But it is there and as you stated, it does not eat something.

I'm just not sure that your test drive was very objective, since you covered only a period of two weeks. It had perhaps ran longer like for a quarter or so to cover the "trends" in spam variations. On the other hand, the high amount of checked mails may compensate this a bit.

Your conclusion is that 100 mails have been caught by using the SPF tests (so without these tests, a mail had scored 4.9 and with SPF tests it came over the "is spam"-score) ? Am I correct in this assumption ?

So why do you see this as a negative result ? You tested for two weeks and got caught 100 mails by enabling a single test (consider that SpamAssassin checks for hundreds in addition).

Well, confessed it is not much, but SPF hasn't been spread as far as it could. Still too unknown to most people. So when more and more webmasters are deciding to implement it, results could get better. But as mentioned before, there is no all-around solution to catch UCE.

By the way, finally we should state the official SPF website for people interesting in getting deeper information from the vendor's perspective: http://www.openspf.org

Although I hate Wikipedia and would not use it as a reference normally, but if there are people wanting to get a short overview together with the main pros/cons and an interesting link list, it is worth having a look: http://en.wikipedia.org/wiki/SenderPolicyFramework

As you stated, we can easily run into evangelistic discussions about SPF (not to forget MS's approach with "Sender ID"), but since we are sharing more or less the same opinion in this topic, there is no real fun doing so 😉
Reply
User profile for user: pterobyte
pterobyte Author
User level: Level 6
11,104 points

Jan 2, 2007 7:20 AM in response to tobias Eichner

You asked for personal opinions/questions, so here
are mine:


I didn't ask for opinions, but this being a public forum I can hardly avoid them can I? 😉

Spam is a huge problem, surely not just for both of
us... for example, I get around (at least roughly
estimated) 500 to 800 spam mails daily in the
meantime; around one year before it was a maximum of
400 on some days, averagely 300. So it grows even
more.


This information is meaningless without at least mentioning what your total mail volume is and without elaborating on what you have done to block spam. If you want to contribute your real world data you are more than welcome. Partial information is not useful and only pollutes the thread.


I'm just not sure that your test drive was very
objective, since you covered only a period of two
weeks. It had perhaps ran longer like for a quarter
or so to cover the "trends" in spam variations. On
the other hand, the high amount of checked mails may
compensate this a bit.


I think a sample of almost half a million mails over 2 weeks, gives a pretty good idea of SPF's effectiveness. Even if running it for a longer time, I am convinced the ratio wouldn't have changed much. Even if from 0.35% it would go up to 1%, in my eyes, it would not justify the network cost.

Your conclusion is that 100 mails have been caught by
using the SPF tests (so without these tests, a mail
had scored 4.9 and with SPF tests it came over the
"is spam"-score) ? Am I correct in this assumption ?


Sort of. Postifx caught about half of those, before involving the content-filter.
For the remaining 50: Yes, but the trigger scores are different on every server.

So why do you see this as a negative result ? You
tested for two weeks and got caught 100 mails by
enabling a single test (consider that SpamAssassin
checks for hundreds in addition).


An average of an extra 100 mails caught per day on an average of 32'000 mails overall!
This means that to catch 0.035% of the 28'000 spam mails I needed 32'000 extra network trips.
Considering that about 20'000 mails were caught without any network test (just postfix checks, bayes and local rules) this is insane (even if a third got cached (unlikely)).

but since we are sharing more or
less the same opinion in this topic.


Don't think we are.
Reply
User profile for user: tobias Eichner
tobias Eichner
User level: Level 3
574 points

Jan 2, 2007 7:55 AM in response to pterobyte

I didn't ask for opinions, but this being a public forum I can hardly avoid them
can I? 😉


Not really. But if you had asked that no one should reply to your thread, I had at least dropped a "Why ?" note 😉

I'm sorry that you consider my provided information being incomplete. But I can't give exact numbers, and this small "me-to" introduction should just state that I deal with the same trouble. It is a proofable fact that I (speak from myself, not clients) get around 800 spam mails daily. I find nothing more being required to know at this time (of course, I could have left this information entirely, but I love long thread replies 😉

I think a sample of almost half a million mails over 2 weeks, gives a pretty
good idea of SPF's effectiveness. Even if running it for a longe


Spammers are quite imaginatively regarding new techniques. So if SPF became more spread, then spammers would not use such domains for spam any longer and more concentrate to domains and mail servers not supporting SPF. This has been simply not considered.

An average of an extra 100 mails caught per day on an average of 32'000
mails overall!


Small steps... at least 🙂

So personally I'm happy with any improvement, may it be as small as this one.

If you want to share this part of your real-world data, how have the SPF tests affected your CPU/memory/network usage ? Significantly ? It seems that you run a large system, maybe even a server cluster ("partial information" are... 😉 and so I would be eager to know how CPU and network usage grows.
Reply
User profile for user: Camelot
Camelot
User level: Level 9
58,575 points

Jan 11, 2007 10:44 AM in response to pterobyte

What might be more interesting is to know the ratio of spam identified using just SPF (or at least SPF first, not last).

In your setup you're using SFP only on the messages that have passed all other checks. Therefore you're not really seeing the amount of spam that SPF identifies. It's possible (althouh I concur unlikely) that 99% of the spam that spamassassin et al. identifies would also be caught by SPF, in which case SPF probably is worth it. SPF just isn't getting a chance to test those messages, though.

As for the resource question, do you think those SPF lookups would consume more resource than running Bayes, SpamAssassin, etc. on 32,000 messages? I can't think so.
As a DNS record, the SPF record would be cached by your host and by your DNS server for whatever time the domain admin sets his TTL (or negative TTL for domains that don't publish a SPF record). Therefore you're going to have one network round trip per domain, not per message.
Compare that with the CPU and time overhead of scanning the entire message content and SPF doesn't look so bad.
Additionally, SPF has a more consistent overhead - one DNS lookup and some perl script compared to content filters whose overhead scales with the size of the message. Granted, I think most spam messages are short and don't tend to have large attachments, but it can be a factor.

Is there any chance you can re-run your test reversing the order of the filters? Given that you're not actually blocking mail that fails the SPF test it shouldn't impact your mail delivery.
Reply
User profile for user: pterobyte
pterobyte Author
User level: Level 6
11,104 points

Jan 11, 2007 11:03 AM in response to Camelot

What might be more interesting is to know the ratio
of spam identified using just SPF (or at least
SPF first, not last).


Let's say I could move it up. I can't move it to the very top, because an SPF pass form a malign domain with SPF record could make me an open relay.

In your setup you're using SFP only on the messages
that have passed all other checks. Therefore you're
not really seeing the amount of spam that SPF
identifies. It's possible (althouh I concur
unlikely) that 99% of the spam that spamassassin et
al. identifies would also be caught by SPF, in which
case SPF probably is worth it. SPF just isn't
getting a chance to test those messages, though.


Agreed! You are potentially right. I doubt the ratio will change significantly, but I am willing to try.

As for the resource question, do you think those SPF
lookups would consume more resource than running
Bayes, SpamAssassin, etc. on 32,000 messages? I can't
think so.


Again, you are right. But only if I could avoid further checks after an SPF pass.
Unfortunately I can't because a published SPF record does not give me the certainety that an mail is clean. Too many spammers, especially in countries which have no legislation regarding spam, have started using SPF records.
So yes, SPF alone doesn't eat more resources on it's own, but since I have to run even passed mail through further checks it is an additiona burden.

Now you could say that I could avoid further checks on SPF fail, but the potential for false positives is too big.

As a DNS record, the SPF record would be cached by
your host and by your DNS server for whatever time
the domain admin sets his TTL (or negative TTL for
domains that don't publish a SPF record). Therefore
you're going to have one network round trip per
domain, not per message.


Agreed. i should have been more precise on this one. The truth is probably somewhere in between. Spam tends however to come a lot from non-existant (even randomly generated) domains, so there still will be quite a few network trips.

I'll have to dig this information out, but I did analyse the spam/different domain ratio some time ago. I can only remember that I was quite surprised. It wasn't like each message was coming from a different domain, but it was still a much higher number of domains than I'd have thought.


Compare that with the CPU and time overhead of
scanning the entire message content and SPF doesn't
look so bad.


Again, only if I could avoid scanning based on SPF results.

Additionally, SPF has a more consistent overhead -
one DNS lookup and some perl script compared to
content filters whose overhead scales with the size
of the message. Granted, I think most spam messages
are short and don't tend to have large attachments,
but it can be a factor.


Sure can be.

Is there any chance you can re-run your test
reversing the order of the filters? Given that you're
not actually blocking mail that fails the SPF test it
shouldn't impact your mail delivery.


Yep. I'd be happy to do this. I'll move SPF up the ladder (as much as I safely can).

Any other info you'd like me to gather while I am at it?

Alex

P.S. Actually, now that I think of it, I could move SPF to the top by modifying the script to log actual results, but always pass on a "DUNNO" to postfix. Not sure if this make sense, because in the real world one would not do this. But for the sake of trying. What do you reckon?
Reply
User profile for user: Camelot
Camelot
User level: Level 9
58,575 points

Jan 11, 2007 11:15 AM in response to pterobyte

I concur on the point of not relying solely on SPF. However, one of SPF's points is that if a domain specifically identifies a set of IPs as being valid and you get a mail from somewhere else it is spam, by defintion. Drop it. Do not pass go. Save your SpamAssassin CPU cycles 🙂

Of course, until SPF is more widely deployed the number of hits in this bucket isn't likely to be high, but it's still a number I'd like to see 🙂

>P.S. Actually, now that I think of it, I could move SPF to the top by modifying the script to log actual results, but always pass on a "DUNNO" to postfix. Not sure if this make sense, because in the real world one would not do this. But for the sake of trying. What do you reckon?

I think this makes sense. Since you're never dropping a message it shouldn't have any impact on the rest of your mail filtering.

What would be most interesting out of this would be the numbers in 4 different buckets:

1) Pass - SPF record exists, and this mail matches
2) Fail - SPF record exists and this mail does not match
3) None - No SPF record
4) Softfail - SPF record too vauge to be sure.

I would expect #3 to be the largest - not enough people publish SPF records, but the ratio of #1:#2 would give an indication as to how effective SPF is when it's implemented.

At the end of the day I don't think SPF is a golden ticket out of spam-****, but I would expect its results to add weight to other filters. For example, I might halve a SpamAssassin score if it comes from a valid SPF-enabled domain.
Reply
User profile for user: pterobyte
pterobyte Author
User level: Level 6
11,104 points

Jan 11, 2007 11:28 AM in response to Camelot

Of course, until SPF is more widely deployed the
number of hits in this bucket isn't likely to be
high, but it's still a number I'd like to see 🙂


What would be most interesting out of this would be
the numbers in 4 different buckets:

1) Pass - SPF record exists, and this mail matches
2) Fail - SPF record exists and this mail does not
match
3) None - No SPF record
4) Softfail - SPF record too vauge to be sure.


Will do. Need a few days before I find time to do this. In order to facilitate the collection of the statistically relevant data, I need to change a few things in the perl script to reflect the "buckets". Then I'll make another 14 days run.

Expect results by the end of the month.

Thanks for your input.

Alex
Reply
User profile for user: pterobyte
pterobyte Author
User level: Level 6
11,104 points

Jan 17, 2007 4:50 AM in response to Camelot

So I did set this up and had it running 5 days for now. I know the 14 days aren't over yet, but I caught a few trends worthwhile mentioning.

The server used is still the same, so all basic data mentioned in my first post still applies.

They main change being that SPF was used as the very first test (before Postfix or SpamAssassin could do their magic). For security reasons I "only" logged results, but had the policy service always return "DUNNO" so that Postfix would further evaluate, rather than trust results from SPF (turns out this was a very good idea, read on).

So let's look at the "buckets" first.
None 65.52 %
Pass 7.18 %
Neutral 22.76 %
Softfail 2.89 %
Fail 1.66 %

As a reminder:
None means no SPF record
Pass and Fail can only be assigned if the SPF record is "restrictive" (-all)
Softfail can only be assigned if the SPF record is a "vague" record (~all)
Neutral can only be assigned if the SPF record is permissive (?all)


So let's start with the obvious bits first.

None:
No SPF record. No action can be taken. SPF is not mandatory and taking action on none would cause a very high number of false positives. So 65.52% of incoming mail was "tested for nothing" and required further checks by Postfix and/or SpamAssassin.

Fail:
We had a 1.66% fail rate. This is the only criteria which would allow for an immediate and complete rejection of an incoming mail. Further checking showed that 99% of mails having a failed spf test would have been stopped by Postfix and/or SpamAssassin (scores were high enough as to not need an SPF score increment). So 1% of 1.66% got caught thanks to SPF.

Softfail:
Interestingly enough, only about 20% was spam. Mostly legit mail being sent through different ISPs.

Pass:
7.18% passed the checking of a restrictive SPF record (-all). On further inspection 40% were spam/viruses coming from servers that had a proper and restrictive SPF record. Didn't check all the domains, but most likely domains situated in countries with no or limited anti-spam laws or spammers "risking" it anyway. Again, SPF alone cannot be trusted. Further inspection is necessary.

Neutral:
Now this in my eyes is the main problem. 22.76% of SPF checks yielded neutral. In other words, yes there is an SPF record, but the domain owner declares that any mail server is allowed to send mail for the domain in question.
85% of mail triggering Neutral was spam!
Yes, spammers have figured out how to use SPF to their advantage. They obviously scan domains for permissive SPF records and then forge those domains.
Now I could just say, neutral is bad and reject it, but the potential for false positives is too high.
Another strategy could be to have SpamAssassin assign a higher score for SPF Neutral, but the potential for false positives remains.


I am very well aware that the test run has been very short so far and I will keep it running, but my feeling is that not much is going to change.

So far, no matter where I place SPF checks, it's effectiveness is close to nil, so why waste resources. At least in my book. I am not going to argue about this, just presenting raw data from one of my servers. Maybe on somebody elses server, they encounter a different reality (maybe somebody wants to do this on another server as well?).

What is worse is that false sense of security this gives to unexperienced sysadmins. So my main advice is, if you are going to use SPF checking, make sure you fully understand it and know how to implement it. Checking in the wrong place of the chain can increase your spam intake and even make you an open relay.

Alex
Reply

This thread has been closed by the system or the community team. You may vote for any posts you find helpful, or search the Community for additional answers.

To SPF or not to SPF

Welcome to Apple Support Community
A forum where Apple customers help each other with their products. Get started with your Apple Account.
Learn more Sign up