The Rules du Jour Spamassassin Tool
There are groups of open source developers that create and update customized spamassassin configuration files that make the tool even more effective. They have even made life easier for the harried systems administrator by creating a script named rules_du_jour that, on a daily basis, will automatically download the rules you select.
The rules_du_jour script can be downloaded from its website at http://www.exit0.us/index.php?pagename=RulesDuJour which has easy to understand installation instructions, but there are some caveats which need to be explained.
The /etc/rulesdujour/config Configuration File
Rules du Jour's configuration file located at /etc/rulesdujour/config has four variables that need to be defined. Each must be enclosed in quotation marks.
The first is SA_DIR, which defines the directory in which you have installed spamassassin. The second is MAIL_ADDRESS which defines the address to which Rules du Jour sends its status messages. The third, SA_RESTART, is the command to be used to restart spamassassin each time the rules_du_jour script is run.
The final parameter, TRUSTED_RULESETS, is the most complicated. It is a space delimited list of all the rules you wish to use. A full list can be found on the Rules du Jour website but there isn't much explanation about what they do and how sensitive each one is to marking email as being spam. Fortunately, you can get this information from the "Rules" section of the Rules Emporium site (http://www.rulesemporium.com/).
It is important to read the notes for each rule they sometimes have sub-groupings of rules that may more suitable for your needs. For example, the SARE_HTML rule includes all the rules in SARE_HTML0, SARE_HTML1, SARE_HTML2, SARE_HTML3 and SARE_HTML_ENG, but according to Rules Emporium, only SARE_HTML0 has a low degree of false positives.
Here is a sample of a /etc/rulesdujour/config configuration file that has taken advantage of some of the more popular and reliable rules.
#
# File: /etc/rulesdujour/config
#
#
# Script information can be found at:
#
# http://www.exit0.us/index.php?pagename=RulesDuJour
#
SA_DIR="/etc/mail/spamassassin"
MAIL_ADDRESS="rulesdujour_update@my-web-site.org"
SA_RESTART="service spamd restart"
TRUSTED_RULESETS="TRIPWIRE SARE_ADULT SARE_OBFU SARE_URI0 SARE_URI1 ANTIDRUG SARE_SPOOF SARE_BAYES_POISON_NXM SARE_OEM SARE_RANDOM SARE_FRAUD SARE_HEADER0 SARE_HEADER2 SARE_HTML0 SARE_SPECIFIC SARE_BML SARE_GENLSUBJ0 SARE_GENLSUBJ2 SARE_WHITELIST"
Note: The Rules du Jour and Rules Emporium sites use the terms "spam" and "ham" frequently. Spam is unwanted email, while ham is the opposite.
Installing Rules du Jour
Installation is fairly simple, here is how it is done:
1) Download the rules_du_jour script with the wget command, make it executable and place it in the /usr/local/bin directory.
[root@bigboy tmp]# wget http://sandgnat.com/rdj/rules_du_jour
--10:58:27-- http://sandgnat.com/rdj/rules_du_jour
=> `rules_du_jour'
Resolving sandgnat.com... 208.42.148.125
Connecting to sandgnat.com
208.42.148.125
:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63,448 (62K) [application/octet-stream]
100%[=======================================>] 63,448 79.85K/s
10:58:28 (79.59 KB/s) - `rules_du_jour' saved [63448/63448]
[root@bigboy tmp]# chmod 700 rules_du_jour
[root@bigboy tmp]# mv rules_du_jour /usr/local/bin
2) Create and edit your /etc/rulesdujour/config configuration file.
[root@bigboy tmp]# mkdir -p /etc/rulesdujour
[root@bigboy tmp]# vi /etc/rulesdujour/config
3) Run the rules_du_jour script, and then run spamassassin in lint mode to test for errors. There should be none.
[root@bigboy tmp]# /usr/local/bin/rules_du_jour
exec: curl -w %{http_code} --compressed -O -R -s -S -z /etc/mail/spamassassin/RulesDuJour/rules_du_jour http://sandgnat.com/rdj/rules_du_jour 2>&1
curl_output: 304
------ TRIPWIRE ------
RULESET_NAME=TRIPWIRE
INDEX=0
CF_URL=http://www.rulesemporium.com/rules/99_FVGT_Tripwire.cf
...
...
...
/^\s*#.*(vers?
version
rev
revision)[:\.\s]*[0-9]/i;'
sort
tail -n 1
CF_MUNGE_SCRIPT=
[root@bigboy tmp]# spamassassin -d --lint
[root@bigboy tmp]#
4) The final step is to add /usr/local/bin/rules_du_jour to your cron table. In this case, I have just created the file /etc/cron.d/rulesdujour with the following entry:
#
# Get latest SpamAssassin rules. Runs at 12:23AM every day.
#
0 23 * * * root /usr/local/bin/rules_du_jour
5) You may have to restart crond to get this configuration to become active
[root@bigboy tmp]# service crond restart
Stopping crond: [ OK ]
Starting crond: [ OK ]
[root@bigboy tmp]#
My experience with Rules du Jour has been very good. Without it, I could only set the required_hits value in the /etc/mail/spamassassin/local.cf file to a relatively insensitive value of 4.5. Anything lower would create too many false positives and valuable mail would be marked as being suspicious. Rules du Jour selectively raises the score of specific types of spam such that I can make the required_hits setting have a very sensitive value of 2.1 with very few false positives.
Using Greylisting
To maximize the effect of their efforts, spammers try to send email as quickly as possible. They take note of the emails that bounce, so that they know which addresses to remove from their lists to make their next mailing more efficient.
When mail servers receive mail too rapidly for them to handle, they can ask the sender to try again later. Spammers often view resending emails to valid addresses as a waste of computing time that could be used to send mail to brand new addresses that belong to faster mail servers. Emails that need to be resent are usually abandoned.
Some emails need reliable delivery to be effective and the senders of these types of messages are willing to resend. These include bank statement notifications, ecommerce purchase confirmations, and subscription newsletters.
In a previous section we saw where spamassassin always rejects emails from blacklisted sources. With greylisting, sources are just asked to resend. One of the most popular greylist mail filter (milter) products is the milter-greylist package which also works seamlessly with spamassassin. It is easy to use and I’ll discuss how can be configured on your mail server.
Downloading and Installing milter-greylist
Installing milter-greylist is relatively simple, but there are a lot of steps. Here’s how to do it:
1. You will have to first install the sendmail-devel software package. Most RedHat and Fedora Linux software products are available in the RPM format. When searching for the RPMs, remember that the filename usually starts with the software package name and is followed by a version number, as in sendmail-devel-8.13.1-2.rpm. (For help downloading, see Chapter 6, "Installing RPM Software").
2. The next step is to download the milter-greylist tar file which you can download from the milter’s website at http://projects.puremagic.com/greylisting/. In this case we download the version 2.0.2 file using the wget command.
[root@bigboy tmp]# wget ftp://ftp.espci.fr/pub/milter-greylist/milter-greylist-2.0.2.tgz
--10:48:06-- ftp://ftp.espci.fr/pub/milter-greylist/milter-greylist-2.0.2.tgz
=> `milter-greylist-2.0.2.tgz'
Resolving ftp.espci.fr... 193.54.82.47
Connecting to ftp.espci.fr
193.54.82.47
:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/milter-greylist ... done.
==> PASV ... done. ==> RETR milter-greylist-2.0.2.tgz ... done.
Length: 116,459 (114K) (unauthoritative)
100%[================================>] 116,459 71.07K/s
10:48:20 (70.79 KB/s) - `milter-greylist-2.0.2.tgz' saved [116459]
[root@bigboy tmp]#
3. Now that you have the file, you’ll have to extract the contents using the tar command.
[root@bigboy tmp]# tar -xzvf milter-greylist-2.0.2.tgz
milter-greylist-2.0.2
milter-greylist-2.0.2/ChangeLog
milter-greylist-2.0.2/Makefile
milter-greylist-2.0.2/Makefile.in
milter-greylist-2.0.2/README
…
…
…
milter-greylist-2.0.2/spf.h
milter-greylist-2.0.2/sync.c
milter-greylist-2.0.2/sync.h
[root@bigboy tmp]#
4. Now enter the newly created milter-greylist directory and compile the package with the configure command. Take a look at the README file before doing this as there may be some additional options you require, but for most cases the defaults are sufficient.
[root@bigboy milter-greylist-2.0.2]# ./configure && make && make install
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
...
...
...
/usr/bin/install -c -d -m 755 /etc/mail
test -f /etc/mail/greylist.conf -o -f /etc/mail/greylist.except
\
/usr/bin/install -c -m 644 greylist.conf /etc/mail
/usr/bin/install -c -d -m 755 -o root /var/milter-greylist
[root@bigboy milter-greylist-2.0.2]#
The next step is to configure the package, which will be covered next.
Configuring milter-greylist
Configuring milter-greylist requires these four quick steps:
1. Add the milter-greylist statements listed in the README file to your /etc/mail/sendmail.mc file:
INPUT_MAIL_FILTER(`greylist',`S=local:/var/milter-greylist/milter-greylist.sock')
define(`confMILTER_MACROS_CONNECT', `j, {if_addr}')
define(`confMILTER_MACROS_HELO', `{verify}, {cert_subject}')
define(`confMILTER_MACROS_ENVFROM', `i, {auth_authen}')
define(`confMILTER_MACROS_ENVRCPT', `{greylist}')
2. There will be a number of shell scripts in the milter-greylist tar directory that correspond to various versions of Linux. Copy the correct version to your /etc/init.d directory and use the chkconfig command to make sure the milter starts on the next reboot.
[root@bigboy milter-greylist-2.0.2]# cp rc-redhat.sh /etc/init.d/milter-greylist
[root@bigboy milter-greylist-2.0.2]# chmod 755 /etc/init.d/milter-greylist
[root@bigboy milter-greylist-2.0.2]# chkconfig --add milter-greylist
[root@bigboy milter-greylist-2.0.2]# chkconfig milter-greylist on
[root@bigboy milter-greylist-2.0.2]# chkconfig --list
grep milter
milter-greylist 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@bigboy milter-greylist-2.0.2]#
3. Edit the /etc/mail/greylist.conf configuration file. Here we set the “try again later” to five minutes and use the whitelist command to deactivate the timer for trusted networks so that mail is delivered immediately.
#
# File: /etc/mail/greylist.conf
#
# How long a client has to wait before we accept
# the messages it retries to send. Here, 1 hour.
#
greylist 5m
#
# Whitelist addresses within my own home/office network
#
acl whitelist addr 192.168.0.0/16
4. Start the milter with the service command.
[root@bigboy milter-greylist-2.0.2]# service milter-greylist start
Starting Milter-Greylist: [ OK ]
[root@bigboy milter-greylist-2.0.2]#
Your new spam mitigation tool should now be fully functional. Delete the mister-greylist directory in /tmp and you should be ready to go!
Configuring milter-greylist
Now that we have milter-greylist installed, we need to be able to do some basic troubleshooting. The /var/log/maillog file should be used to determine what is happening to your mail. Here are two samples of what to expect:
Dec 24 00:32:31 bigboy sendmail[28847]: jBO8WVnG028847: Milter: to=,
reject=451 4.7.1 Greylisting in action, please come back in 00:05:00
Dec 23 20:40:21 bigboy milter-greylist: jBO4eF2m027418: addr 211.115.216.225 from
rcpt : autowhitelisted for 24:00:00
In the first entry, the email received is given a tag (jBO8WVnG028847) based on key characteristics in the mail header and a request is sent to the sender to resend the email in five minutes. Any email that is received with the same calculated key within the autowhite period configured in the greylist.conf file will then be automatically accepted without delay. In the second entry, the email has been resent and immediately accepted. Any other email from that source within the next 24 hours will be accepted without delay.
Note: Greylisting is very effective, but you will have to tne its operation to make sure critical emails are not delayed at all. One soluton is to set the autowhite period in /etc/mail/greylist.conf to slightly more than 24 hours especially if you get mail from certain recipients, such as newsletters, on a daily basis. This makes them arrive without interruption.
There are groups of open source developers that create and update customized spamassassin configuration files that make the tool even more effective. They have even made life easier for the harried systems administrator by creating a script named rules_du_jour that, on a daily basis, will automatically download the rules you select.
The rules_du_jour script can be downloaded from its website at http://www.exit0.us/index.php?pagename=RulesDuJour which has easy to understand installation instructions, but there are some caveats which need to be explained.
The /etc/rulesdujour/config Configuration File
Rules du Jour's configuration file located at /etc/rulesdujour/config has four variables that need to be defined. Each must be enclosed in quotation marks.
The first is SA_DIR, which defines the directory in which you have installed spamassassin. The second is MAIL_ADDRESS which defines the address to which Rules du Jour sends its status messages. The third, SA_RESTART, is the command to be used to restart spamassassin each time the rules_du_jour script is run.
The final parameter, TRUSTED_RULESETS, is the most complicated. It is a space delimited list of all the rules you wish to use. A full list can be found on the Rules du Jour website but there isn't much explanation about what they do and how sensitive each one is to marking email as being spam. Fortunately, you can get this information from the "Rules" section of the Rules Emporium site (http://www.rulesemporium.com/).
It is important to read the notes for each rule they sometimes have sub-groupings of rules that may more suitable for your needs. For example, the SARE_HTML rule includes all the rules in SARE_HTML0, SARE_HTML1, SARE_HTML2, SARE_HTML3 and SARE_HTML_ENG, but according to Rules Emporium, only SARE_HTML0 has a low degree of false positives.
Here is a sample of a /etc/rulesdujour/config configuration file that has taken advantage of some of the more popular and reliable rules.
#
# File: /etc/rulesdujour/config
#
#
# Script information can be found at:
#
# http://www.exit0.us/index.php?pagename=RulesDuJour
#
SA_DIR="/etc/mail/spamassassin"
MAIL_ADDRESS="rulesdujour_update@my-web-site.org"
SA_RESTART="service spamd restart"
TRUSTED_RULESETS="TRIPWIRE SARE_ADULT SARE_OBFU SARE_URI0 SARE_URI1 ANTIDRUG SARE_SPOOF SARE_BAYES_POISON_NXM SARE_OEM SARE_RANDOM SARE_FRAUD SARE_HEADER0 SARE_HEADER2 SARE_HTML0 SARE_SPECIFIC SARE_BML SARE_GENLSUBJ0 SARE_GENLSUBJ2 SARE_WHITELIST"
Note: The Rules du Jour and Rules Emporium sites use the terms "spam" and "ham" frequently. Spam is unwanted email, while ham is the opposite.
Installing Rules du Jour
Installation is fairly simple, here is how it is done:
1) Download the rules_du_jour script with the wget command, make it executable and place it in the /usr/local/bin directory.
[root@bigboy tmp]# wget http://sandgnat.com/rdj/rules_du_jour
--10:58:27-- http://sandgnat.com/rdj/rules_du_jour
=> `rules_du_jour'
Resolving sandgnat.com... 208.42.148.125
Connecting to sandgnat.com
208.42.148.125
:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63,448 (62K) [application/octet-stream]
100%[=======================================>] 63,448 79.85K/s
10:58:28 (79.59 KB/s) - `rules_du_jour' saved [63448/63448]
[root@bigboy tmp]# chmod 700 rules_du_jour
[root@bigboy tmp]# mv rules_du_jour /usr/local/bin
2) Create and edit your /etc/rulesdujour/config configuration file.
[root@bigboy tmp]# mkdir -p /etc/rulesdujour
[root@bigboy tmp]# vi /etc/rulesdujour/config
3) Run the rules_du_jour script, and then run spamassassin in lint mode to test for errors. There should be none.
[root@bigboy tmp]# /usr/local/bin/rules_du_jour
exec: curl -w %{http_code} --compressed -O -R -s -S -z /etc/mail/spamassassin/RulesDuJour/rules_du_jour http://sandgnat.com/rdj/rules_du_jour 2>&1
curl_output: 304
------ TRIPWIRE ------
RULESET_NAME=TRIPWIRE
INDEX=0
CF_URL=http://www.rulesemporium.com/rules/99_FVGT_Tripwire.cf
...
...
...
/^\s*#.*(vers?
version
rev
revision)[:\.\s]*[0-9]/i;'
sort
tail -n 1
CF_MUNGE_SCRIPT=
[root@bigboy tmp]# spamassassin -d --lint
[root@bigboy tmp]#
4) The final step is to add /usr/local/bin/rules_du_jour to your cron table. In this case, I have just created the file /etc/cron.d/rulesdujour with the following entry:
#
# Get latest SpamAssassin rules. Runs at 12:23AM every day.
#
0 23 * * * root /usr/local/bin/rules_du_jour
5) You may have to restart crond to get this configuration to become active
[root@bigboy tmp]# service crond restart
Stopping crond: [ OK ]
Starting crond: [ OK ]
[root@bigboy tmp]#
My experience with Rules du Jour has been very good. Without it, I could only set the required_hits value in the /etc/mail/spamassassin/local.cf file to a relatively insensitive value of 4.5. Anything lower would create too many false positives and valuable mail would be marked as being suspicious. Rules du Jour selectively raises the score of specific types of spam such that I can make the required_hits setting have a very sensitive value of 2.1 with very few false positives.
Using Greylisting
To maximize the effect of their efforts, spammers try to send email as quickly as possible. They take note of the emails that bounce, so that they know which addresses to remove from their lists to make their next mailing more efficient.
When mail servers receive mail too rapidly for them to handle, they can ask the sender to try again later. Spammers often view resending emails to valid addresses as a waste of computing time that could be used to send mail to brand new addresses that belong to faster mail servers. Emails that need to be resent are usually abandoned.
Some emails need reliable delivery to be effective and the senders of these types of messages are willing to resend. These include bank statement notifications, ecommerce purchase confirmations, and subscription newsletters.
In a previous section we saw where spamassassin always rejects emails from blacklisted sources. With greylisting, sources are just asked to resend. One of the most popular greylist mail filter (milter) products is the milter-greylist package which also works seamlessly with spamassassin. It is easy to use and I’ll discuss how can be configured on your mail server.
Downloading and Installing milter-greylist
Installing milter-greylist is relatively simple, but there are a lot of steps. Here’s how to do it:
1. You will have to first install the sendmail-devel software package. Most RedHat and Fedora Linux software products are available in the RPM format. When searching for the RPMs, remember that the filename usually starts with the software package name and is followed by a version number, as in sendmail-devel-8.13.1-2.rpm. (For help downloading, see Chapter 6, "Installing RPM Software").
2. The next step is to download the milter-greylist tar file which you can download from the milter’s website at http://projects.puremagic.com/greylisting/. In this case we download the version 2.0.2 file using the wget command.
[root@bigboy tmp]# wget ftp://ftp.espci.fr/pub/milter-greylist/milter-greylist-2.0.2.tgz
--10:48:06-- ftp://ftp.espci.fr/pub/milter-greylist/milter-greylist-2.0.2.tgz
=> `milter-greylist-2.0.2.tgz'
Resolving ftp.espci.fr... 193.54.82.47
Connecting to ftp.espci.fr
193.54.82.47
:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done. ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/milter-greylist ... done.
==> PASV ... done. ==> RETR milter-greylist-2.0.2.tgz ... done.
Length: 116,459 (114K) (unauthoritative)
100%[================================>] 116,459 71.07K/s
10:48:20 (70.79 KB/s) - `milter-greylist-2.0.2.tgz' saved [116459]
[root@bigboy tmp]#
3. Now that you have the file, you’ll have to extract the contents using the tar command.
[root@bigboy tmp]# tar -xzvf milter-greylist-2.0.2.tgz
milter-greylist-2.0.2
milter-greylist-2.0.2/ChangeLog
milter-greylist-2.0.2/Makefile
milter-greylist-2.0.2/Makefile.in
milter-greylist-2.0.2/README
…
…
…
milter-greylist-2.0.2/spf.h
milter-greylist-2.0.2/sync.c
milter-greylist-2.0.2/sync.h
[root@bigboy tmp]#
4. Now enter the newly created milter-greylist directory and compile the package with the configure command. Take a look at the README file before doing this as there may be some additional options you require, but for most cases the defaults are sufficient.
[root@bigboy milter-greylist-2.0.2]# ./configure && make && make install
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
...
...
...
/usr/bin/install -c -d -m 755 /etc/mail
test -f /etc/mail/greylist.conf -o -f /etc/mail/greylist.except
\
/usr/bin/install -c -m 644 greylist.conf /etc/mail
/usr/bin/install -c -d -m 755 -o root /var/milter-greylist
[root@bigboy milter-greylist-2.0.2]#
The next step is to configure the package, which will be covered next.
Configuring milter-greylist
Configuring milter-greylist requires these four quick steps:
1. Add the milter-greylist statements listed in the README file to your /etc/mail/sendmail.mc file:
INPUT_MAIL_FILTER(`greylist',`S=local:/var/milter-greylist/milter-greylist.sock')
define(`confMILTER_MACROS_CONNECT', `j, {if_addr}')
define(`confMILTER_MACROS_HELO', `{verify}, {cert_subject}')
define(`confMILTER_MACROS_ENVFROM', `i, {auth_authen}')
define(`confMILTER_MACROS_ENVRCPT', `{greylist}')
2. There will be a number of shell scripts in the milter-greylist tar directory that correspond to various versions of Linux. Copy the correct version to your /etc/init.d directory and use the chkconfig command to make sure the milter starts on the next reboot.
[root@bigboy milter-greylist-2.0.2]# cp rc-redhat.sh /etc/init.d/milter-greylist
[root@bigboy milter-greylist-2.0.2]# chmod 755 /etc/init.d/milter-greylist
[root@bigboy milter-greylist-2.0.2]# chkconfig --add milter-greylist
[root@bigboy milter-greylist-2.0.2]# chkconfig milter-greylist on
[root@bigboy milter-greylist-2.0.2]# chkconfig --list
grep milter
milter-greylist 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@bigboy milter-greylist-2.0.2]#
3. Edit the /etc/mail/greylist.conf configuration file. Here we set the “try again later” to five minutes and use the whitelist command to deactivate the timer for trusted networks so that mail is delivered immediately.
#
# File: /etc/mail/greylist.conf
#
# How long a client has to wait before we accept
# the messages it retries to send. Here, 1 hour.
#
greylist 5m
#
# Whitelist addresses within my own home/office network
#
acl whitelist addr 192.168.0.0/16
4. Start the milter with the service command.
[root@bigboy milter-greylist-2.0.2]# service milter-greylist start
Starting Milter-Greylist: [ OK ]
[root@bigboy milter-greylist-2.0.2]#
Your new spam mitigation tool should now be fully functional. Delete the mister-greylist directory in /tmp and you should be ready to go!
Configuring milter-greylist
Now that we have milter-greylist installed, we need to be able to do some basic troubleshooting. The /var/log/maillog file should be used to determine what is happening to your mail. Here are two samples of what to expect:
Dec 24 00:32:31 bigboy sendmail[28847]: jBO8WVnG028847: Milter: to=
reject=451 4.7.1 Greylisting in action, please come back in 00:05:00
Dec 23 20:40:21 bigboy milter-greylist: jBO4eF2m027418: addr 211.115.216.225 from
In the first entry, the email received is given a tag (jBO8WVnG028847) based on key characteristics in the mail header and a request is sent to the sender to resend the email in five minutes. Any email that is received with the same calculated key within the autowhite period configured in the greylist.conf file will then be automatically accepted without delay. In the second entry, the email has been resent and immediately accepted. Any other email from that source within the next 24 hours will be accepted without delay.
Note: Greylisting is very effective, but you will have to tne its operation to make sure critical emails are not delayed at all. One soluton is to set the autowhite period in /etc/mail/greylist.conf to slightly more than 24 hours especially if you get mail from certain recipients, such as newsletters, on a daily basis. This makes them arrive without interruption.
No comments:
Post a Comment