Greylisting on Red Hat 9
Last updated, Feb 9, 2005
Jan Labanowski  jkl -[at]- ccl.net

I am running a Web site and a listserver (http://www.ccl.net
and spam is a problem beyond imagination. The list operates
since 1991, and the e-mail addresses associated with this
mail exploder are know to any !@&# spammer on the earth.
By the way, read about "Spam Kings". Brian McWilliams
wrote a fascinating book about them:
http://www.oreilly.com/catalog/spamkings/. 
To learn about spam on the technical level, get the recent
book by B. Costales and M. Flynt -- "sendmail Milters: A Guide
for Fighting Spam" (http://spambook.bcx.org/).
See also: http://www.milter.org/milter_api/,
http://www.samag.com/documents/s=7178/sam0206l/0206l.htm,
and http://sourceforge.net/docman/display_doc.php?docid=141&group_id=8173

While I will be moving soon to a more recent versions of Linux than RH9
for my list server, I have installed the Greylisting (BTW, this is how
it is spelled) to see if it works, and it WORKS GREAT. But there are
problem too...

What is Greylisting? In short, it is a method of defeating spam (and
virus). The idea is ingenious. Legitimate electronic mail is distributed by
Mail Transfer Agents (MTAs). Most popular of them is sendmail. The sendmail
contacts your mail gateway (Mail Exchanger, MX) when it wants to pass mail
to you. When your MX is up, it takes in the mail and passes it to your
mailbox (often using some other program, Mail Delivery Agent, MDA, like
procmail). Then, you look at your mail using your Mail User Agent (MUA), which
shows you the message, allows you to respond to it, delete, save it, etc.

What if your mail gateway (MX -- Mail Exchanger) is down or too busy to
accept mail? The senders MTA queues the mail, and tries to deliver it
later. In most cases, it tries again within few minutes, and then, if
still "No Go", it will try at lower frequency in an hour, few hours, etc,
until it decides that your MX is really dead, and then it will bounce
the mail back to the sender's mailbox and you see it in your
MUA (usually 4 days). 

Now, spammers do not send spam like this. They use special software
(spamware/ratware) that sends mail as soon as possible, without paying
attention to the status codes reported by recipient MX. They deliver
millions of pieces of mail in few hours and there is no time to 
analyze and react accordingly. It would not be cost effective
to queue undelivered messages and do it the way "good people"
do it. If they cannot deliver the spam message to a given
recipient, they go to the next one on the list. In fact that do
not usually know if the mail was delivered since it is cheaper to
send e-mail to a bouncing address than it is to maintain their
list of addresses to spam. They use many machines (spam-engines)
with different IP addresses, put random From: and To: addresses
in message header (that is why it does not even make sense to
bounce or respond to spam mail since the From: address
is phony). At this time, they almost never resend the spam message
when it is not delivered the first time, at least not from
the same IP address, and not with the same sender's address
on the From: line. If they did, then there are professional
services that collect spam (like: honeypots -- machines with
phony usernames that attract spammers but which would never be
contacted for legitimate reasons), from people who send samples
of spam as "good citizens" to abuse@some.ISP.address, from their
spam monitoring software, etc). These services extract the sender's
IP addresses from the spam messages, and put them into a database
in real time (within minutes or even seconds). These databases are
shared with subscribers (unfortunately, good things do not
come for free, and if you want professional service, you need
to pay for it). Then subscribers to these services block the
the spam mail from these IP addresses on their MX machines.
It is an oversimplified picture, but it is the main idea...

Quick note on addresses. The To: and From: addresses that you
see in the e-mail message header ARE NOT USED at the time of
delivery. They are just part of the e-mail message -- like
with real mail, they are inside the envelope, and they are not
used by the post office to deliver mail. The mail is delivered via
Simple Mail Transfer Protocol (SMTP) where the sender's MTU
(Mail Transfer Agent, mail relay) contacts your MX (Mail Exchanger)
and tells your MX its Envelope From:, and the Envelope To: address
(i.e. recipient of the mail). The sender's MTU also provides
its IP address since it is a TCP connection and they have to
exchange their Internet identities (IP addresses).
While the Envelope From: may be bogus or even Null (if they
do not want to get a bounce back, in case of nondelivery)
the IP address is usually real (it is VERY hard to use fake
source IP address, since during SMTP dialog both machines have
to talk to each other and you can only talk back to a real IP
address -- though even this can be sometimes circumvented when
border routers are not well guarded). The Envelope To: address
has to be real, or you would not get the mail.

What would happen if you did not accept the message from
the given sender from the given machine on the first try,
saying: "I am busy, try later" ? You would only save the
identity, but temporarily reject the message. Then, on the
next try, say, after an hour, you would accept the message
from the sender. The good chance it that for spam, you
would never get a retry from the spammer, at least not with
the same sender address and from the same machine. They
not only do not run real MTAs that batch undelivered messages
for future retries, but they will usually not send the same spam
from the same machine, since the Blacklisting databases
put them on the Reject list before they have a chance to
send a repeat.

This is the main idea behind greylisting. It does not
come without the price though. WE ARE DELAYING E-MAIL!!!.
But there are good news too. You can WHITELIST the
senders who are "Good People", and also, the greylist
software whitelists for a period of time the senders
who came through (i.e, who retried and succeeded in
delivering the message to your MX machine). Moreover,
the greylisting software automatically whitelists the
people to whom you sent mail (and CCL sends mail to
its subscribers everyday), and therefore replies to
your mail message come to you without delay. In short,
in the CCL/mail-list environment, the delay is usually
only on the first message ever seen from the particular
address. But there are many other issues and problems,
not excluding the fact that some senders (MTUs)
do not adhere to the standard, and do not resend
mail which was temporarily rejected. Moreover, some
organizations receive mail (MX servers) on different
machines than they sent (relays, gateways).

You have to understand these issues THOROUGHLY before
you even attempt to install greylisting on your machine.
Frankly, I am not sure if the greylisting is a good idea
for a institutional mail exchanger, but there are ways
to make it better, and I am currently reading about the
pitfalls and workaround. There is also honeypots that I am
exploring -- I have a score of addresses (former students
who left, for example) that only receive spam -- no legitimate
email. These can be used to blacklist IP address on first
try. But I need to experiment with the blacklist period.
In my case, the mail exchanger dedicated to a specific task,
and I am also providing an alternative means of sending
mail to CCL via the Web submission form for CCL members
who have problems with my greylisting setup and their
nonstandard relays. 

If you think about greylisting but you do not have access
to the ROOT account on your mail exchanger, do not even
read further. This stuff works ONLY on the Mail Exchanger,
before mail is saved in your personal mailbox.
You cannot do greylisting on your local desktop,
unless it is also your mail exchanger.
It will not work when you forward mail from your
"institutional" mail exchanger to your local
computer, since your "institutional" mail exchanger
will batch the mail if your local machine refuses it,
and will keep resending it, even if it is a spam.
Again, it only works on the MX which is a primary
recipient of the mail.

Now, read the following pages. All my credit goes to these
people who wrote them. Details are in the links below,
and I only list here my own approach, and underscore
some points that I had problems understanding due to my
limited mental capacity.

http://projects.puremagic.com/greylisting/links.html

http://projects.puremagic.com/greylisting/index.html

http://www.greylisting.org/implementations/

http://www.maynidea.com/sendmail/relaydelay.html

There are other greylisting milters, maybe even more sophisticated,
but I installed relaydelay for the time being, and I work on expending
it (it is in perl, so testing is easy).

You need to be a root and install few packages before you fire
up your greylist. I was working on the Red Hat 9 Linux machine,
with EVERYTHING installed, including of course perl and mysql.
If you have only partial RedHat installation, you may need
additional packages, and possibly some backward compatibility
libraries.

Install few perl modules. You can be in some temp directory.
I was in a /root/relaydelay directory usually.

a) install perl DBI
perl -MCPAN -e 'install DBI'

b) install perl DBM::mysql
perl -MCPAN -e 'get DBD::mysql'
cd /root/.cpan/build/DBD-mysql-2.9004
perl Makefile.PL --testpassword='ThePswd' --testuser='root'
make install

c) install perl NET::Deamon. I had actually a problem here.
   It is probably because my machine is a 2-processor SMP.
   The test for forkm does not work.
       perl -MCPAN -e 'install Net::Daemon'
   t/forkm.........  (seats there forever and then:
        FAILED tests 1-10
   make: *** [test_dynamic] Error 29
   /usr/bin/make test -- NOT OK
   Running make install
   make test had returned bad status, won't install without force

   So I just cheated and did not do make test, went to
       http://www.cpan.org/modules/by-module/Net/
       and grabbed Net-Daemon release:
       wget http://www.cpan.org/modules/by-module/Net/Net-Daemon-0.38.tar.gz
       tar zxvf Net-Daemon-0.38.tar.gz
       cd Net-Daemon-0.38
       perl Makefile.PL
       make
       make install

d) Install Sendmail::Milter
     perl -MCPAN -e 'get Sendmail::Milter'
     cd /root/.cpan/build/Sendmail-Milter-0.18
     perl Makefile.PL /usr /usr/lib
     make install

Then you need to deal with MySQL database which should be
installed on your system. If you never used it, you need to
activate the start-up on boot.

Edit /etc/rc.d/init.d/mysqld and make sure you have a line like:
# chkconfig: 345 78 12

Then do:
chkconfig --del mysqld
chkconfig --add mysqld
chkconfig --list mysqld

then start mysql server.

/etc/rc.d/init.d/mysqld start

Note... it is unsafe (runs without password enabled), but we will
get to it shortly. There is a virus out there which attacks
mysql, so make sure access is secured by password.

Download the relaydelay software

mkdir /root/relaydelay
cd /root/relaydelay
wget http://projects.puremagic.com/greylisting/releases/relaydelay-0.04.tgz
tar zxvf relaydelay-0.04.tgz
cd /root/relaydelay/relaydelay-0.04


Before you do anything with relaydelay, download the patch from:
The original post is at:
http://lists.puremagic.com/pipermail/greylist-users/2003-September/000261.html

The patch itself:
cd /root/relaydelay/relaydelay-0.04
wget http://lists.puremagic.com/pipermail/greylist-users/attachments/20030904/b8dafed9/relaydelay-0.04.bin

Then patch the files with the relaydelay-0.04.bin

cd /root/relaydelay/relaydelay-0.04
patch -p1 < relaydelay-0.04.bin

It patched files: relaydelay.conf relaydelay.pl and
creates a new file: relaydelay (startup script)

Edit file mysql.sql and change line 23 to be

grant select,insert,update,delete on relaydelay.* to spam@'localhost' identified by 'MyHardPassword';

PLEASE!!! use something else for password than MyHardPassword.

This will later on set the password for the spam database user
(note, database users are totally unrelated to UNIX users in
/etc/password -- they are separate users used only within mysql
database server)

Change the password of the root user password for the database
(again, it has nothing to do with root UNIX user, it is mysql root).
You need to do it only if you had never set password for mysql root.
It comes usually unprotected and you need to do it before
you can use it sensibly and securely.

mysql -u root
Then I set the password by typing a command at mysql> prompt.
NOTE, do not forget about ";" at the end. 

mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('Not-A-Unix-Root-Password!');
Query OK, 0 rows affected (0.07 sec)
mysql> exit
Bye

Then check if mysql ate what you gave it to it:

mysql -u root -p
Enter password: Not-A-Unix-Root-Password!

mysql> exit


Now you are ready to create the database tables and user for
relaydelay software (you will be asked for the mysql root password
(here I used as an example the: Not-A-Unix-Root-Password!).

cd /root/relaydelay/relaydelay-0.04
mysql -u root -p < mysql.sql

You can now check if it worked. Log in as user spam (with the user
spam password, (here, MyHardPassword which you set in mysql.sql).

mysql -u spam -p
mysql> use relaydelay;
mysql> show tables;
+----------------------+
| Tables_in_relaydelay |
+----------------------+
| dns_name             |
| relayreport          |
| relaytofrom          |
+----------------------+
3 rows in set (0.00 sec)

mysql> exit;
Remember what you used for the password for the spam
mysql user (in the example above it was MyHardPassword
but you used something else on line 23, did you?).

Then you need to do some editing.

Edited the db_maintenance.pl and replaced
my $database_user = 'db_user';
my $database_pass = 'db_pass';
 with
my $database_user = 'spam';
my $database_pass = 'MyHardPassword';

copied db_maintenance.pl to /usr/sbin
cp /root/relaydelay/relaydelay-0.04/db_maintenance.pl /usr/sbin
chmod 700 /usr/sbin/db_maintenance.pl

edited /root/relaydelay/relaydelay-0.04/relaydelay.conf
and changed lines:
$database_user = 'db_user';
$database_pass = 'db_pass';

to lines

$database_user = 'spam';
$database_pass = 'MyHardPassword';

also changed:
$pass_mail_when_db_unavail = 0;
to 
$pass_mail_when_db_unavail = 1;

(I rarely log in to this machine, and I chose to
pass, rather than reject mail when mysql died.).

copied /root/relaydelay/relaydelay-0.04/relaydelay.conf
to /etc/mail/relaydelay.conf

chmod 700 /etc/mail/relaydelay.conf

ran the /usr/sbin/db_maintenance.pl
/usr/sbin/db_maintenance.pl

It worked...

Added /usr/sbin/db_maintenance.pl to cron

crontab -e

Added line (run db_maintenance.pl on Saturdays at 2:24am).

     24   2   *   *   6    /usr/sbin/db_maintenance.pl


(note, the /var/spool/cron/root file may be set to
read only, so you need to make sure to set permissions
and a write for root if your editor refuses to save it).

ran the comment
    crontab -l

to make sure it made it there.

Edited the relaydelay.pl in /root/relaydelay/relaydelay-0.04 
and corrected the following lines:

my $database_user = 'spam';
my $database_pass = 'MyHardPassword';

also
my $log_file = '';
to
my $log_file = '/var/log/relaydelay.log';

cp relaydelay.pl /usr/sbin
chmod 700 /usr/sbin/relaydelay.pl


Edited the script relaydelay on /root/relaydelay/relaydelay-0.04
that was created by the patch. Changed:
/usr/local/sbin/relaydelay.pl $CONFIG
   to
/usr/sbin/relaydelay.pl $CONFIG

copied the script to /etc/rc.d/init.d/
cp /root/relaydelay/relaydelay-0.04/relaydelay /etc/rc.d/init.d/
chmod 700 /etc/rc.d/init.d/relaydelay

To make sure that it starts on reboot I did:
cd /etc/rc.d/init.d
chkconfig --del relaydelay
chkconfig --add relaydelay
chkconfig --list relaydelay
The last command told me that it starts on runlevel 2 3 4 5:
relaydelay      0:off   1:off   2:on    3:on    4:on    5:on    6:off

Then I started the relaydelay milter:
/etc/rc.d/init.d/relaydelay start

ps -ef | grep relay
root     23158     1  0 01:09 ?        00:00:00 /usr/bin/perl -w /usr/sbin/relaydelay.pl /etc/mail/relaydelay.conf

checked also the /var/log/relaydelay.log
which contained this text:
----------
Loaded Config File: /etc/mail/relaydelay.conf
Using connection 'local:/var/run/relaydelay.sock' for filter relaydelay
DBI Connecting to DBI:mysql:database=relaydelay:host=localhost:port=3306
Spawned relaydelay daemon process 32298.
Starting Sendmail::Milter 0.18 engine.
---------

time to edit /etc/mail/sendmail.mc

I already run the spamassassin milter (my installation essentially
followed the page:
http://www.eschew.net/geek/spamassassin.php

(with few minor changes, which I will describe soon).
So I have to deal with 2 milters. (If you do not have another
milter, you need only to put one line into /etc/mail/sendmail.mc:
INPUT_MAIL_FILTER(`relaydelay', `S=local:/var/run/relaydelay.sock,T=S:1m;R:2m;E:3m')dnl
)


Since I have two mail filters now in my sendmail.mc file I had to put there:

INPUT_MAIL_FILTER(`relaydelay', `S=local:/var/run/relaydelay.sock,T=S:1m;R:2m;E:3m')dnl
INPUT_MAIL_FILTER(`spamassassin', `S=local:/var/run/spamass.sock,F=,T=C:15m;S:4m;R:4m;E:10m')dnl
define(`confINPUT_MAIL_FILTERS',`relaydelay,spamassassi')dnl

This tells sendmail to call relaydelay first, and then to call
spamasssassin (it does not make much sense to check mail for
spam if it can be rejected at the next step -- but --  it probably does
not matter much, since relaydelay rejects mail usually before the DATA stage,
and spamassassin chews on DATA, i.e., complete mail message).

Then, to make these changes available to sendmail via sendmail.mc
you need to run m4 preprocessor. Check the top of your sendmail.mc
file for instructions, in my case:

cd /etc/mail
make -C /etc/mail
(you can also do it as:
m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf
)

It rebuilds the sendmail.cf. Check its modification time to see if
it was actually rebuilt.

Since now I need to run sendmail and bunch of other things
I created a script /usr/local/bin/restart_sendmail
---------------
#!/bin/bash

/etc/rc.d/init.d/spamassassin stop
/etc/rc.d/init.d/spamass-milter stop
/etc/rc.d/init.d/relaydelay stop
/etc/rc.d/init.d/sendmail stop

/etc/rc.d/init.d/sendmail start
/etc/rc.d/init.d/relaydelay start
/etc/rc.d/init.d/spamass-milter start
/etc/rc.d/init.d/spamassassin start

-------------
(again, if you do not run spamassassin milter, just skip
the appropriate lines that contain spamassassin and spammass-milter).

chmod 755 /usr/local/bin/restart_sendmail
restart_sendmail


From my experience with spamass-milter, it sometimes dies.
Probably spammers put some stuff into messages which makes it
die. Most likely some new version of spamassassin fixes it,
but I run a pretty old version spamassassin-2.55-3.

I also created 
/usr/local/bin/restart_sendmail_if_dead.pl
This script checks if all daemons are running, and if one
of them is dead, restarts sendmail and friends.
It is not elegant, but it works:
------------
#!/usr/bin/perl

open(PS, "/bin/ps -ef | ") || die "Cannot open ps\n";

$no_spamd = 1;
$no_spamass_milter = 1;
$no_relaydelay = 1;
$no_sendmail = 1;
$date = localtime();

while ($line = <PS>) {
    if($line =~ /\/usr\/bin\/spamd/) {
	print STDERR "spamd running at $date\n";
	$no_spamd = 0;
    }
    if($line =~ /spamass-milter/) {
	print STDERR "spamass-milter running at $date\n";
	$no_spamass_milter = 0;
    }
    if($line =~ /sendmail:\s+accepting\s+connections/) {
	print STDERR "sendmail running at $date\n";
	$no_sendmail = 0;
    }
    if($line =~ /relaydelay/) {
	print STDERR "relaydelay running at $date\n";
	$no_relaydelay = 0;
    }

}
if($no_spamd || $no_spamass_milter || $no_sendmail || $no_relaydelay) {
    print STDERR "Restarting sendmail and spammassassin\n";
    system("/etc/rc.d/init.d/spamassassin stop");
    system("/etc/rc.d/init.d/spamass-milter stop");
    system("/etc/rc.d/init.d/relaydelay stop");
    system("/etc/rc.d/init.d/sendmail stop");

    system("/etc/rc.d/init.d/sendmail start"); 
    system("/etc/rc.d/init.d/relaydelay start");
    system("/etc/rc.d/init.d/spamassassin start");
    system("/etc/rc.d/init.d/spamass-milter start");
}
-----------------------

I had put it to the crontab, to be run every few hours.


I used the scripts at:
http://lists.puremagic.com/pipermail/greylist-users/2003-November/000327.html
(or http://www.maynidea.com/sendmail/addlist.html )
to create whitelists and blacklists and add them to the relaydelay
database.

Cut and Pasted and created the file: addlist.pl
and edited it slightly
cd /root/relaydelay/relaydelay-0.04

Since I used patched relaydelay.conf I used my beloved
emacs editor and added variables:
my $run_as_daemon;
my $log_file;

then
chmod 755 addlist.pl
cp -p addlist.pl /usr/local/bin


Then I used:
dig domain MX 
dig hostname
to find the addresses of friendly MX to put into a whitelist.
(For example:
dig nd.edu MX
pickering.cc.nd.edu
)
HOWEVER... The Mail Exchangers (MX) are hosts that RECEIVE the mail.
While in many cases, they are the same as MAIL RELAYS (GATEWAYS)
for outgoing mail, they need not be the same. For many installations
especially large, they are not the same. There is a new protocol
(SPF -- Sender Policy Framework, http://spf.pobox.com/execsumm.html )
that provides the DNS entries which announce the outgoing SMTP servers
as a special TXT record in the DNS, but it is still not widely adopted.
So, for the time being, I put to a whitelist the addresses of MXs for friendly
hosts, and the IP addresses that I scavenge from my incoming mail from
Received: headers, and put them to whitelist_ip.txt
beside those given with relaydelay distribution.

129.74 			# nd.edu
192.148			# ccl.net
192.88			# oar.net
192.146			# osu.edu
192.74.137		# theworld.com
67.28.113		# yahoo.com
64.157.4.78
67.28.114.36
64.156.215
67.28.114
66.218.86
68.142.202

I also created a blacklist_ip.txt file to zap the bad guys.

Then I ran these in my install directory /root/relaydelay/relaydelay-0.04:
cd /root/relaydelay/relaydelay-0.04
./addlist.pl -whitelist 9999-12-31 23:59:59 < whitelist_ip.txt
./addlist.pl -blacklist 9999-12-31 23:59:59 < blacklist_ip.txt

I will most likely modify these scripts to do some checking ("Is this
thing already in relaydelay database?"), and some scripts to extract
the bad and good guys IPs from my mailboxes looking at the Received:
header lines.

The relaydelay.log file will grow fast, and you need to rotate it
(i.e., move it periodically as
mv relaydelay.log relaydelay.log.1
but before you do it, you do:
for n=n_max to 2
  mv relaydelay.log.n-1 -> relaydelay.log.n;
There is a rotate service in Linux, and what you have
to do, is to create the following file (relaydelay)
in the /etc/logrotate.d directory:

# rotate relaydelay.log (see /etc/rc.d/init.d/relaydelay service)
#
/var/log/relaydelay.log {
        notifempty
	daily
        rotate 9
        missingok
    postrotate
       # it is probably a good idea to restart this stuff daily
       /usr/local/bin/restart_sendmail
    endscript
}

The cron will run this script once a day.