Tuning spam defence

Question

Level 2

195 points

Tuning spam defence

HI to all,
i need help tuning my mail system on OsX Server 10.4.11 about some particular spam messages that my box can't reject.
I followed all procedures written on osx.topicdesk.com paper "Frontline..." and 80% of spam messages are rejected but since 2-3 months (i live in Italy) the situation become different, no more spam in russian or chinese language, no more sex related (easily rejected by spamassassin rules or via RBL's) but a huge mass mailings about bank offers or related. The language used in these mails is italian (better than my english :-D) , html inside it
doesn't hide scripts or tricked codes, it seems a real and true mail message.
Whole of them coming from united states or england and the 80% of them report something like this:
------------
Content-Type: text/html; charset="Windows-1251"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-Msmail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2600.0000
X-Mimeole: Produced By Microsoft MimeOLE V6.00.2600.0000
Message-Id: <SERVERsyGh4DGSwiINI0000066e@ipg.uk.com>
X-Originalarrivaltime: 02 Oct 2008 12:03:08.0218 (UTC) FILETIME=[D574A9A0:01C92486]
X-Virus-Scanned: by amavisd-new at localhost
X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char E0 hex) in message header 'Subject': Subject: ...premia il suo account con un bonus di fedelt\340\n
X-Spam-Status: No, hits=0.181 tagged_above=-999 required=4 tests=AWL
X-Spam-Level:
------------------------

I noticed that X-Spam-Level is blank so spamassassin didn't find anything wrong (maybe it was true because the content is clean)
Is there a way to block these mail for their Bad Header or 7bit Encoding? Could be a good idea adding postgrey to reject them?
I post also my postconf in case of needed modifications or suggestions..
----------------------
alias_maps = hash:/etc/aliases
command_directory = /usr/sbin
config_directory = /etc/postfix
content_filter = smtp-amavis:[127.0.0.1]:10024
daemon_directory = /usr/libexec/postfix
debug peerlevel = 2
disable vrfycommand = yes
enable serveroptions = yes
header_checks = regexp:/etc/postfix/header_checks
html_directory = no
inet_interfaces = all
mail_owner = postfix
mailbox sizelimit = 0
mailbox_transport = cyrus
mailq_path = /usr/bin/mailq
manpage_directory = /usr/share/man
maps rbldomains =
message sizelimit = 0
mydestination = $myhostname,localhost.$mydomain,localhost,xxxxxx.it,mail.xxxxxxxxx.it
mydomain = xxxxxxxxx.it
mydomain_fallback = localhost
myhostname = mail.xxxxxxxxx.it
mynetworks = 127.0.0.1/32,192.168.0.0/24
mynetworks_style = host
newaliases_path = /usr/bin/newaliases
owner requestspecial = no
queue_directory = /private/var/spool/postfix
readme_directory = /usr/share/doc/postfix
recipient_delimiter = +
sample_directory = /usr/share/doc/postfix/examples
sendmail_path = /usr/sbin/sendmail
setgid_group = postdrop
smtpd clientrestrictions = permit saslauthenticated permit_mynetworks reject rblclient zen.spamhaus.org reject rblclient bl.spamcop.net reject rblclient dnsbl.sorbs.net reject rblclient psbl.surriel.com reject rblclient dnsbl.njabl.org permit
smtpd datarestrictions = permit_mynetworks, reject unauthpipelining, permit
smtpd helorequired = yes
smtpd helorestrictions = permit saslauthenticated, permit_mynetworks, check heloaccess hash:/etc/postfix/helo_access, reject non_fqdnhostname, reject invalidhostname, permit
smtpd pw_server_securityoptions = login,cram-md5,plain
smtpd recipientrestrictions = reject invalidhostname, reject non_fqdnsender, reject non_fqdnrecipient, permit sasl_authenticated,permit_mynetworks,reject_unauth_destination,reject_rblclient zen.spamhaus.org, permit
smtpd sasl_authenable = yes
smtpd tls_keyfile =
smtpd use_pwserver = yes
unknown local_recipient_rejectcode = 550
-------------------------------

thank you!

MBP 2.2GHz-C2D, 45 macs in lan, Mac OS X (10.5.2), 3 Gb Ram

Posted on Oct 2, 2008 5:48 AM

Reply

Answer 1

pterobyte

Level 6

11,104 points

Oct 2, 2008 9:15 AM in response to myriapod

Since those mails make it through even though you have a properly configured Postfix setup, chances are they are sent from "well behaved" servers. Thus, chances are, that greylisting won't help much either.

The next level would be to use more aggressive content level filtering. This can be done by adding rules to SpamAssassin. You can do this manually ( http://www.rulesemporium.com/rules.htm), or by using spamtrainer and issuing "spamtrainer -a" ( http://osx.topicdesk.com/spamtrainer/).

Failing that, you will need to analyze those mails and depending on patterns either write your own rules or see if you can blacklist the senders.

HTH,
Alex

Reply

Answer 2

myriapod Author

Level 2

195 points

Oct 2, 2008 9:59 AM in response to pterobyte

hi pterobyte i was waiting for your technical support!
I already use .cf rules of course (installed via spamtrainer -a), the server is running since 5 years ago
and i think i was one of the first person using your fantastic tool when it was released , but as i wrote before, i can't realize how to reject those annoying mails i this period...
Simply maximizing or making spam filters more aggressive (now it's set on 4) for me isn't a good method , i tried some days ago and i received many false positive 70% coming from chickenpox.cf wrong tagging.
Now i'm trying to edit chickenpox.cf giving less score and using meta scoring, i will try in this case to set to 5 or 6 the junkfilters value.
And what about "BAD HEADER" in amavisd-alert tag? Many of spam messages report this issue
and , i don't know if this was the cause, all of them show No Hits in Spam Tags...
Can you suggest me a whitepaper about writing a spamassassin rule?

thank you for your support!

Reply

Answer 3

pterobyte

Level 6

11,104 points

Oct 3, 2008 12:49 AM in response to myriapod

Unfortunately, there are too many legit mails with bad headers to be able to act on it.

See this for writing rules: http://wiki.apache.org/spamassassin/WritingRules

Also, do not edit chickenpox.cf itself to adjust scores. Rather create a seperate file for all your score adjustments. For example, create /etc/mail/spamassassin/z_myscores.cf and in it add:
score FB WORD1_ENDDOLLAR 0.5
score FB YOURMORTGAGE 0.5
...
When done, save and restart amavisd.

Reply

Answer 4

myriapod Author

Level 2

195 points

Oct 10, 2008 7:16 AM in response to pterobyte

can you suggest me a valid AntiSpam Firewall Hardware?
I'm taking a look at Barracuda spam firewall (100 users model) it seems interesting...

Reply

Answer 5

pterobyte

Level 6

11,104 points

Oct 10, 2008 8:17 AM in response to myriapod

The Barracuda boxes work well. Other than that, you will need to decide based on your needs, which box to go for.

There isn't much you couldn't do yourself with amavisd, spamassassin and ClamAV, but it certainly takes more work than just getting a Barracuda and pay them to keep it updated.

Reply

Answer 6

myriapod Author

Level 2

195 points

Oct 10, 2008 10:37 AM in response to pterobyte

my last question please...
I enabled network test in spamassassin (in local.cf) e try to add via cpan Net::DNS
but it hangs while installing via cpan on something related to FTP:
this is my output

root# cpan -i Net::DNS
CPAN: Storable loaded ok
Going to read /private/var/root/.cpan/Metadata
Database was generated on Wed, 08 Oct 2008 19:28:23 GMT
LWP not available
CPAN: Net::FTP loaded ok
Fetching with Net::FTP:
ftp://**.********.**/****/*******/********.**.gz

<Edited by Moderator>

---------
then hangs here....

If i try to make spamassassin test an email saved on my destop with
spamassassin -tD < path to message
it reports sa debug version infos/.cf rules (1 error on DNS ) and at the end it doesn't show
its default email spam statistic about what catched by .cf rules....

Any idea to fix this?

i report sa debug output (what do you think?is it ok?):

---------
89-96-213-240:/Users/admin root# spamassassin -D --lint
debug: SpamAssassin version 3.0.1
debug: Score set 0 chosen.
debug: running in taint mode? yes
debug: Running in taint mode, removing unsafe env vars, and resetting PATH
debug: PATH included '/bin', keeping.
debug: PATH included '/sbin', keeping.
debug: PATH included '/usr/bin', keeping.
debug: PATH included '/usr/sbin', keeping.
debug: Final PATH set to: /bin:/sbin:/usr/bin:/usr/sbin
debug: diag: module not installed: DBI ('require' failed)
debug: diag: module installed: DB_File, version 1.810
debug: diag: module installed: Digest::SHA1, version 2.10
debug: diag: module installed: IO::Socket::UNIX, version 1.21
debug: diag: module installed: MIME::Base64, version 3.05
debug: diag: module not installed: Net::DNS ('require' failed)
debug: diag: module not installed: Net::LDAP ('require' failed)
debug: diag: module not installed: Razor2::Client::Agent ('require' failed)
debug: diag: module installed: Storable, version 2.13
debug: diag: module installed: URI, version 1.35
debug: ignore: using a test message to lint rules
debug: using "/etc/mail/spamassassin/init.pre" for site rules init.pre
debug: config: read file /etc/mail/spamassassin/init.pre
debug: using "//usr/share/spamassassin" for default rules dir
debug: config: read file //usr/share/spamassassin/60_whitelist.cf
debug: using "//etc/mail/spamassassin" for site rules dir
debug: config: read file //etc/mail/spamassassin/00 FVGTFile001.cf
debug: config: read file //etc/mail/spamassassin/70 sareadult.cf
debug: config: read file //etc/mail/spamassassin/70 saregenlsubj0.cf
debug: config: read file //etc/mail/spamassassin/70 sareheader0.cf
debug: config: read file //etc/mail/spamassassin/70 sarehtml0.cf
debug: config: read file //etc/mail/spamassassin/70 sareobfu0.cf
debug: config: read file //etc/mail/spamassassin/70 sareoem.cf
debug: config: read file //etc/mail/spamassassin/70 sarespoof.cf
debug: config: read file //etc/mail/spamassassin/70 sarestocks.cf
debug: config: read file //etc/mail/spamassassin/70 sareunsub.cf
debug: config: read file //etc/mail/spamassassin/70 sarewhitelist.cf
debug: config: read file //etc/mail/spamassassin/71 sare_redirectpre3.0.0.cf
debug: config: read file //etc/mail/spamassassin/72 sare_redirectpost3.0.0.cf
debug: config: read file //etc/mail/spamassassin/backhair.cf
debug: config: read file //etc/mail/spamassassin/bogus-virus-warnings.cf
debug: config: read file //etc/mail/spamassassin/edizioniambiente.cf
debug: config: read file //etc/mail/spamassassin/local.cf
debug: config: read file //etc/mail/spamassassin/omar.cf
debug: config: read file //etc/mail/spamassassin/weeds.cf
debug: using "/private/var/root/.spamassassin" for user state dir
debug: using "/private/var/root/.spamassassin/user_prefs" for user prefs file
debug: config: read file /private/var/root/.spamassassin/user_prefs
debug: plugin: loading Mail::SpamAssassin::Plugin::URIDNSBL from @INC
debug: failed to load Net::DNS::Resolver: Can't locate Net/DNS.pm in @INC (@INC contains: //lib/perl5/site_perl /System/Library/Perl/5.8.6/darwin-thread-multi-2level /System/Library/Perl/5.8.6 /Library/Perl/5.8.6/darwin-thread-multi-2level /Library/Perl/5.8.6 /Library/Perl /Network/Library/Perl/5.8.6/darwin-thread-multi-2level /Network/Library/Perl/5.8.6 /Network/Library/Perl /System/Library/Perl/Extras/5.8.6/darwin-thread-multi-2level /System/Library/Perl/Extras/5.8.6 /Library/Perl/5.8.1) at /System/Library/Perl/Extras/5.8.6/Mail/SpamAssassin/Plugin/URIDNSBL.pm line 113.

debug: plugin: registered Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x1b75150)
debug: plugin: loading Mail::SpamAssassin::Plugin::Hashcash from @INC
debug: plugin: registered Mail::SpamAssassin::Plugin::Hashcash=HASH(0x1afd3e0)
debug: plugin: loading Mail::SpamAssassin::Plugin::SPF from @INC
debug: plugin: registered Mail::SpamAssassin::Plugin::SPF=HASH(0x1afe40c)
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x1b75150) implements 'parse_config'
debug: plugin: Mail::SpamAssassin::Plugin::Hashcash=HASH(0x1afd3e0) implements 'parse_config'
config: SpamAssassin failed to parse line, skipping: auto_learn 1
config: SpamAssassin failed to parse line, skipping: safe_reporting 0
config: SpamAssassin failed to parse line, skipping: use tersereport 0
config: SpamAssassin failed to parse line, skipping: subject_tag * Warning: Junk Mail *
config: SpamAssassin failed to parse line, skipping: rewrite_subject 0
warning: description for FS STARTDOYOU2 is over 50 chars
debug: using "/private/var/root/.spamassassin" for user state dir
debug: bayes: 12392 tie-ing to DB file R/O /private/var/root/.spamassassin/bayes_toks
debug: bayes: 12392 tie-ing to DB file R/O /private/var/root/.spamassassin/bayes_seen
debug: bayes: found bayes db version 3
debug: using "/private/var/root/.spamassassin" for user state dir
debug: bayes: Not available for scanning, only 0 spam(s) in Bayes DB < 200
debug: bayes: 12392 untie-ing
debug: bayes: 12392 untie-ing db_toks
debug: bayes: 12392 untie-ing db_seen
debug: Score set 1 chosen.
debug: ---- MIME PARSER START ----
debug: main message type: text/plain
debug: parsing normal part
debug: added part, type: text/plain
debug: ---- MIME PARSER END ----
debug: bayes: 12392 tie-ing to DB file R/O /private/var/root/.spamassassin/bayes_toks
debug: bayes: 12392 tie-ing to DB file R/O /private/var/root/.spamassassin/bayes_seen
debug: bayes: found bayes db version 3
debug: bayes: Not available for scanning, only 0 spam(s) in Bayes DB < 200
debug: bayes: 12392 untie-ing
debug: bayes: 12392 untie-ing db_toks
debug: bayes: 12392 untie-ing db_seen
debug: metadata: X-Spam-Relays-Trusted:
debug: metadata: X-Spam-Relays-Untrusted:
debug: decoding: no encoding detected
debug: Loading languages file...
debug: Language possibly: en,sco
debug: metadata: X-Languages: en sco
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x1b75150) implements 'parsed_metadata'
debug: Running tests for priority: 0
debug: running header regexp tests; score so far=0
debug: all '*From' addrs: ignore@compiling.spamassassin.taint.org
debug: all '*To' addrs:
debug: running body-text per-line regexp tests; score so far=0
debug: running uri tests; score so far=0
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x1b75150) implements 'check_tick'
debug: running raw-body-text per-line regexp tests; score so far=0
debug: running full-text regexp tests; score so far=0
debug: Running tests for priority: 500
debug: plugin: Mail::SpamAssassin::Plugin::URIDNSBL=HASH(0x1b75150) implements 'check postdnsbl'
debug: running meta tests; score so far=0
debug: running header regexp tests; score so far=0
debug: running body-text per-line regexp tests; score so far=0
debug: running uri tests; score so far=0
debug: running raw-body-text per-line regexp tests; score so far=0
debug: running full-text regexp tests; score so far=0
debug: Running tests for priority: 1000
debug: running meta tests; score so far=0
debug: running header regexp tests; score so far=0
debug: using "/private/var/root/.spamassassin" for user state dir
debug: lock: 12392 created /private/var/root/.spamassassin/auto-whitelist.lock.89-96-213-240.ip14.fastwebn et.it.12392
debug: lock: 12392 trying to get lock on /private/var/root/.spamassassin/auto-whitelist with 0 retries
debug: lock: 12392 link to /private/var/root/.spamassassin/auto-whitelist.lock: link ok
debug: Tie-ing to DB file R/W in /private/var/root/.spamassassin/auto-whitelist
debug: auto-whitelist (db-based): ignore@compiling.spamassassin.taint.org|ip=none scores 0/0
debug: AWL active, pre-score: 0, autolearn score: 0, mean: undef, IP: undef
debug: DB addr list: untie-ing and unlocking.
debug: DB addr list: file locked, breaking lock.
debug: unlock: 12392 unlink /private/var/root/.spamassassin/auto-whitelist.lock
debug: Post AWL score: 0
debug: running body-text per-line regexp tests; score so far=0
debug: running uri tests; score so far=0
debug: running raw-body-text per-line regexp tests; score so far=0
debug: running full-text regexp tests; score so far=0
debug: is spam? score=0 required=4
debug: tests=
debug: subtests= _SARE_WHITELISTFLAG
lint: 6 issues detected. please rerun with debug enabled for more information.
-------------------

Reply

Answer 7

pterobyte

Level 6

11,104 points

Oct 10, 2008 9:48 AM in response to myriapod

The errors of the debug output are purely cosmetic and not relevant (those parameters are overwritten by amavisd anyway).

Regarding the CPAN installation, it can take a very long time to get the mirrors the first time you use CPAN (even more than 15 minutes). If it still hangs, add other mirrors to your CPAN configuration:
http://www.cpan.org/misc/cpan-faq.html
http://mirrors.cpan.org/

Also, once installed, you need to enable network tests in amavisd.conf not local.cf

Message was edited by: pterobyte

Reply

Answer 8

myriapod Author

Level 2

195 points

Oct 10, 2008 11:18 AM in response to pterobyte

i solved hangs on cpan typing : export ftp_passive=1 on terminal
but after cpan retriving sources needed (.gz) and its dependencies
it reports an error on "make" telling that " install seems impossible" so DNS module
can't be installed...
Everything starts as follows:

Writing Makefile for Net::IP
-- NOT OK
Running make test
Can't test without successful make
Running make install
make had returned bad status, install seems impossible

this happened when building IP module...

About local.cf you're right i was confused , i enabled network tests on amavisd.conf.

Reply

Answer 9

pterobyte

Level 6

11,104 points

Oct 10, 2008 11:43 AM in response to myriapod

Are Developer Tools (XCode) installed? If not, you must do so first.

If yes, try this:
sudo perl -MCPAN -e shell

then at the CPAN prompt:
o conf prerequisites_policy follow
force install Net::DNS

Reply

Answer 10

myriapod Author

Level 2

195 points

Oct 11, 2008 5:10 AM in response to pterobyte

of course i got Developer Tools :-D

try to reset Cpan settings , delete also /Users/myaccount/.cpan
Update Cpan shell to latest version but nothing new ,"Make" failed for Net::IP
and Digest::HMAC_MD5.
Should this issue linked to their particular version?
I will try to fetch again manually and edit MakeFile.pl maybe the answer is here..

Reply

Answer 11

pterobyte

Level 6

11,104 points

Oct 11, 2008 5:53 AM in response to myriapod

You shouldn't need to edit Makefile.pl.

Look at the output of CPAN, it whoudl tell you why the prerequisites fail to compile.
You can try and force install those as well.

Reply

Answer 12

myriapod Author

Level 2

195 points

Oct 13, 2008 5:32 AM in response to pterobyte

ok , module Net::DNS installed after manually make Net::IP and Digest::HMAC_MD5...
I enabled network test in amavisd.conf and try to
spamassassin -tD < path to spam message
is it normal an error at the end of output : "no report template found" ?
Enabling network test means that it try to resolve DNS queries for every messagge to be scanned?

Reply

Answer 13

pterobyte

Level 6

11,104 points

Oct 13, 2008 5:35 AM in response to myriapod

When you run spamassassin manually, you must make sure you run it under user clamav.
That said, just look at headers of caught spam and you will see if your network tests work.

Yes, enabling network tests, means that spamasassin will do network tests for each mail message. DNS queries will be cached.

Reply

Answer 14

myriapod Author

Level 2

195 points

Oct 13, 2008 5:53 AM in response to pterobyte

what about turning on also AWL in amavis?
i saw in amavis.conf AWL is not enabled..

#$sa autowhitelist = 1; # turn on AWL (default: false)

thank you for your support again..

Reply

Answer 15

pterobyte

Level 6

11,104 points

Oct 13, 2008 5:55 AM in response to myriapod

This is only relevant for older spamassassin version. If you want to start adding more features, I recommend you update spamassassin and amavisd to the latest versions.

Reply