Historical Background

There's got to be a better way.

I've been working on Courier, in some form or fashion, since September of 1997.  This project didn't start out anywhere close to where it ended up being.  Courier began its life as a handful of completely separate an unrelated projects, that, over time, managed to stick to each other.

The roots of Courier go back to early 1997, when I became frustrated with procmail. It's a nice tool that does its job adequately, but it shows its age. At that time, procmail did not support maildirs, and I was running Qmail and wanted to keep my mail in maildirs.  Furthermore, procmail's mail filtering specifications were, and still are, somewhat archaic and hard to understand.  There's got to be a better way.

Additionally, I've reached a conclusion that the mail filtering was not being properly implemented in the first place. Mail servers accepted all mail without discrimination, and mail filters are applied after messages are already accepted and locally spooled.

Unwanted junk mail proliferated in the early 1990s primarily because it placed so little burden on the senders, shifting its burden and costs on the recipients, who then have to deal with mailboxes stuffed with junk, and having to wade through all that flotsam in order to find the scant few non-junk messages.

Although post-delivery mail filtering could help somewhat, this was far from being an ideal situation. If you set the mail filter to autodelete unwanted mail, false positive indications will result in legitimate mail being lost forever.  Filing apparently unwanted mail away means that you still have to waste time emptying out your junk folder, looking for any misfiled legitimate messages.  You haven't really accomplished anything, you just postponed the inevitable until some later time.  Or, you can also set the mail filter to return the unwanted mail back to its sender, on the theory that the sender of a message that was mistakenly blocked will notify you later that your mail filter has malfunctioned. Unfortunately, most of unwanted junk mail has a forged return address, and the forged victim almost always ends up dealing with a distributed denial of service attack, as mail servers all across the Internet quickly dump megabytes of returned mail in his mailbox. That cannot be avoided, but by returning blocked mail to sender, you've unwittingly participated in the denial of service attack.

It was obvious that the correct way to block unwanted mail is to refuse to accept the mail from the sending mail server in the first place. I've put together a couple of hacks that had Qmail invoke procmail before accepting the message from the sender.  It worked, sort of, but it wasn't pretty. Qmail was easy to work with, but something like that simply wasn't that either Qmail or procmail was designed for. Add that to the fact that procmail could not deliver to maildirs, at that time, when used in its traditional mail filtering role, and the fact that the syntax of its filtering language was not very intuitive, and this particular solution quickly lost its appeal.

And that's how maildrop came about.  Its original concept was merely a better procmail, that had a more or less modern programming language, that didn't do some of the stupid things that procmail did (such as reading an entire 5 megabyte message into RAM), and could unload mail into maildirs natively. But, I also written maildrop to have a separate "embedded mode", that allowed a far lesser Qmail hack than with procmail. And, for over a year, this worked well enough to leave it alone, except for a few occasional tweaks to maildrop's programming language.

Still, although I now had a fairly decent mail filter that talks to maildirs, reading mail from maildirs was still somewhat troublesome. My favorite mail client, Pine, cannot access maildirs. There are some patches floating that add maildir capability to Pine, but they provide only a partial solution, and Pine still has to keep its mail folders as mailbox files, with all of their usual blemishes and warts.  Plus, every time I upgraded the operating system on my workstation, I had to reapply the Pine maildir patches.  There's got to be a better way.

The better way that I've settled on was to write a maildir-reading webmail CGI, SqWebMail. SqWebMail came to life in December 1998/January 1999. It read and wrote mail directly to maildirs, kept all of its folders as maildirs, and was extremely tiny but still provided a great lot of features.  Going into late 1999 I worked on a spell checker, Ispell, and added extensive MIME support. I was able to reuse a suprising amount of code. It might seem that the two applications have nothing in common.  Yet, both have to deal with mail headers, and MIME constructs, and maildrop had a fully modular RFC 822 and RFC 2045 parsing library. That's almost a third of the code in SqWebMail right there (not anymore, actually, but it was true in the beginning).

SqWebMail was great, and I still use it quite often.  However, there are times that I find Pine to be much more comfortable.  I don't always find a web browser interface to be a convenient way to read or compose mail, so I still longed for Pine.  I suddenly realized that all I really need to do is to have Pine talk to a maildir-friendly IMAP server.  Pine is a great IMAP client, so by putting all folders, including INBOX, on an IMAP server on the same machine, that uses maildirs, would allow Pine to keep all of its mail in maildirs, without having to fuss around with patches.

Unfortunately, there was no such beast in existence. There were no available IMAP servers that supported maildirs. There was a patch for the UW-IMAP server, in fact, it's the same patch as the one for Pine, since both of them use the same mail access library, and we'd be back to square one.  There's got to be a better way.

Fine, so off I go to write an IMAP client.  The project broke ground in late spring 1999, but there were other things happening at the same time.

Around the same time I was becoming slowly dissatisfied with Qmail. It was a raw, efficient, mail relay, but it lacked a lot of features, in fact besides raw SMTP delivery, it had very little else to show for itself. It couldn't do much more than deliver a single message, to a single recipient, on a single remote mail server, in a single connection.  It was not capable of batching all recipient for the same domain in the same delivery.  When it had a message with 100 recipients at domain.com, it would dutifully open a connection to domain.com's server, send a single message to a single recipient, close the connection, then lather, rinse, and repeat 99 more times.  It would do all of these deliveries in parallel, true, up to the outbound concurrency limit, but it was still wasted a lot of bandwidth and time.  Qmail could not open a single connection, and send just one message addressed to all 100 recipients. Qmail did not support delivery status notifications, and did not properly handle 8bit MIME mail -- it would send 8bit MIME mail to a mail server even if it did not advertise support for 8bit MIME mail. Additionally, Qmail's design forced the server to accept all mail for any recipient with a local domain address, and bounce mail for nonexistent local recipients internally.  Qmail was not capable of rejecting mail addressed to nonexistent local mailboxes.  I wrote some patches to fix that, but that was merely another hack.  Additionally, DJB didn't seem to be actively developing Qmail anymore.  There were occasional mentions of the next version of Qmail, over the course of a couple of years, but very little to show for it.  There were also some licensing issues with Qmail that some people did not like.  There's got to be a better way.

So, I decided to tighten my belt by a few notches, and write a mail server, almost simultaneously with the decision to write a quick little IMAP server so I can continue to play with Pine. I've reached a belief that simply none of the existing mail transport and access software out there was to my liking. So, it was time to do have a better way to do it.  I picked "Courier" for the name of the mail server, so, for a lack of an imagination, the IMAP server became known as "Courier-IMAP". I announced publicly that I was working on Courier-IMAP, but kept Courier pretty much to myself.

I completed Courier-IMAP in September of 1999, almost at the same time I completed a functional version of Courier to the extent that I was able to remove Qmail and switch over to Courier.  I initially used Courier for a couple of weeks for my outgoing mail, keeping Qmail around to handle incoming mail, then finally bit the bullet, and removed Qmail completely after it became obvious that the monster was alive and kicking.

It might seem a bit strange that it took the same amount of time to write a small IMAP server, and an entire mail server, but it really isn't. For starters, I already had a large pile of code as a base.  A lot of mail handling from both maildrop, and SqWebMail was easily recycled into Courier and Courier-IMAP.  Additional code that went into Courier-IMAP was immediately copied into Courier, and some of it even found its way back into further enhancements to SqWebMail (and maildrop, to a much lesser extent).  Additionally, for the first time in this project, I actually drew up some kind of a design, before I started to hack! Both maildrop, SqWebMail, and Courier-IMAP, came completely off the top of my head.  But, before writing a single line of Courier code, I actually sat down, and drew a high level flowchart of Courier's mail server specification! I coded the mail server core straight from the specs, and amazingly enough it worked right the first time. Since September 1999, the amount of tweaking to the Courier mail server core was completely superficial.

Those specs ended up saving me a lot of time, and they are included in the source code.  They have very little use now, make very little sense to anyone except me, but they're there, as a historical record, of some kind.

The rest of the work since then involved all the piggyback modular code - outbound mail code, and inbound mail code. Also, keep in mind that the first version was rather plain, all it could do was a basic send/receive mail via ESMTP.  It did not have any kind of a mail filtering API -- which was slightly amusing because that's how the whole thing got started.  It did not have some of the more exotic authentication code (vpopmail, LDAP, and several others), it did not have a UUCP gateway, or SOCKS proxying capability. Plus, maildrop, SqWebmail, and Courier-IMAP were still standalone projects, and were not yet integrated into the same source code. That's the work that's been going on since then.

PREV: Links NEXT: Courier