Administering A Mail Server

Contents

Contents

References

http://www.axllent.org/docs/networking/gmail_pop3_with_fetchmail

http://download.gna.org/hpr/fetchmail/FAQ/gmail-pop-howto.html

http://ubuntuforums.org/showthread.php?t=565326

http://www.runpcrun.com/google-pop3-access

Fetchmail

Fetchmail Multidrop Issues

"If you're using multidrop, you always want to be using 'envelope'"

Collecting email from a mailhost where each person has their own POP3 mailbox is straight forward, the email address of the recipient is reliably contained in the 'To:' field. Collecting email from a multidrop / wildcard / catchall mailbox is problematic because of the different methods the mailhost uses to designate individual recipients when they're all lumped together in the one mailbox.

In setting up email collection with Fetchmail we must test for a few scenarios where email collection can potentially fail.

To help you work through these problems you can have Fetchmail leave the mail on the server then use a web mail interface if one is available or a separate client program such as Thunderbird, or a dedicated POP3 mailbox reader utility. (Be careful not to let the mail pile up enough that it exceeds your disk quota at the mail host though).

It would be a good idea when you first setup a mail server not to delete from the mail host. Let it stack up and see if you have any problems such as those discussed here. If you're having problems with list mail it is easier to look in the mail on the upstream mail host for any list mail, then look if it got delivered to the people, rather than wait for them to tell you later that list mail isnt arriving.

This is a good description of the problem: Requisites for working multidrop mailboxes by Matthias Andree - http://www.dt.e-technik.uni-dortmund.de/~ma/mail/multidrop

When multiple addresses at your domain in the To: field, people receive multiple emails

Where an email is sent to multiple people at your domain, the 'To:' field of each email might contain both peoples' addresses. Thus the 'To:' field doesn't represent the individual to whom a particular email is intended, it represents all of them. There will be an email in the POP3 mailbox for each person, each with these same addresses in the 'To:' field, rather than one email addressed to everyone, or one per person with only them in the 'To:' field. If Fetchmail were to process the 'To:' field in this case it would give a copy to each person and they would receive multiple copies of the same email.

The mail host or mail host software is allowed to choose one of a number of available header fields to put the relevant recipient in. Some mailhosts won't even use the correct fields. You have to find which alternate mail header field contains just one recipient and tell Fetchmail to use that.

To test for this: send an email to two people at your domain at once; look in the POP3 mailbox; there will be two emails there; diff the two and to see which field is being used to differentiate between recipients. Here are some examples:

Example 1 - 'X-Original-To' and 'Delivered-To'

Section of header from email (a):

To: richard@example.org.uk, alan@example.org.uk
X-Original-To: richard@example.org.uk
Delivered-To: richard@example.org.uk

Section of header from email (b):

To: richard@example.org.uk, alan@example.org.uk
X-Original-To: alan@example.org.uk
Delivered-To: alan@example.org.uk

Fetchmail is processing 'To:' when it should be processing 'X-Original-To' or 'Delivered-To'.

Example 2 - 'Delivered-To'

Fetchmail defaults to checking 'X-Envelope-To'. You may not have 'X-Envelope-To' so need to instruct Fetchmail to instead check 'Delivered-To:' by using 'envelope "Delivered-To:"'.

Example 3 - 'Received'

Section of header from email (c) (showing only the last of multiple 'Received:' fields):

To: david@exampledomain.co.uk, vee@exampledomain.co.uk
Received: by yw-out-1718.google.com with SMTP id 5so356542ywr.30 for <david@exampledomain.co.uk>; Sun, 01 Jun 2008 06:15:29 -0700 (PDT)

Section of header from email (d) (showing only the last of multiple 'Received:' fields):

To: david@exampledomain.co.uk, vee@exampledomain.co.uk
Received: by yw-out-1718.google.com with SMTP id 5so356543ywr.30 for <vee@exampledomain.co.uk>; Sun, 01 Jun 2008 06:15:29 -0700 (PDT)

Here the mail host isn't inserting 'X-Original-To' or 'Delivered-To' fields, but we find the address we're looking for in 'Received: ... for'.

Each mail server says where the mail came from in delivering to it in 'Received:' fields, starting lower down the header, each mail server adding to the header as you read up.

If you telnet into the mailhost on port 110 you'll see what software they're using. This is with 'MailSite POP3 Server' software. You can research a particular mail server software or mailhost organisation's practices by doing a web search such as 'mailsite pop3 server catchall fetchmail' and you may be fortunate to see people discussing your particular situation.

'Received:' line parsing issue with multiple possible mailhosts operating on the mail

If you're using 'Received:' there's an issue to beware of.

"Now fetchmail is complaining that "thmailsite4.services.byworkwise.com is not an alias of the mailserver" but why should it be?"
"If fetchmail had only never introduced Received line parsing. You can use "aka" to list aliases."

"The 'aka' option is for use with multidrop mailboxes. It allows you to pre-declare a list of DNS aliases for a server. This is an optimization hack that allows you to trade space for speed. When fetchmail, while processing a multidrop mailbox, grovels through message headers looking for names of the mailserver, pre-declaring common ones can save it from having to do DNS lookups. Note: the names you give as arguments to 'aka' are matched as suffixes -- if you specify (say) 'aka netaxs.com', this will match not just a hostnamed netaxs.com, but any hostname that ends with '.netaxs.com'; such as (say) pop3.netaxs.com and mail.netaxs.com."

Background Reading

"Your setup is inherently broken and cannot be fixed until the moment where your provider stuffs one copy per recipient and keeps the original envelope recipient into some header."

"What about fetchmail Received line parsing?

"-E <line> | --envelope <line> (Keyword: envelope) This option changes the header fetchmail assumes will carry a copy of the mail's envelope address. Normally this is 'X-Envelope-To' but as this header is not standard, practice varies. See the discussion of multidrop address handling below. As a special case, 'envelope "Received"' enables parsing of sendmail-style Received lines. This is the default, and it should not be necessary unless you have globally disabled Received parsing with 'no envelope' in the .fetchmailrc file.

In the .fetchmailrc file, the 'envelope' string argument may be preceded by a whitespace-separated number. This number, if specified, is the number of such headers to skip (that is, an argument of 1 selects the second header of the given type). This is sometime useful for ignoring bogus Received headers created by an ISP's local delivery agent.

When there is more than one local name (or name mapping) the fetchmail code does look at the Received, To, Cc, and Bcc headers of retrieved mail (this is 'multidrop mode'). It looks for addresses with hostname parts that match your poll name or your 'via', 'aka' or 'localdomains' options, and usually also for hostname parts which DNS tells it are aliases of the mailserver. See the discussion of 'dns', 'checkalias', 'localdomains', and 'aka' for details on how matching addresses are handled."

From the Fetchmail FAQ: "Users are getting multiple copies of messages.

It's a consequence of multidrop. What's happening is that you have N users subscribed to the same list. The list software sends N copies, not knowing they will end up in the same multidrop box. Since they are both locally addressed to all N users, fetchmail delivers N copies to each user.

Fetchmail tries to eliminate adjacent duplicate messages in a multidrop mailbox. However, this logic depends on the message-ID being identical in both copies. It also depends on the two copies being adjacent in the server mailbox. The former is usually the case, but the latter condition sometimes fails in a timing-dependent way if the server was processing multiple incoming mail streams.

I could eliminate this problem by keeping a list of all message-IDs received during a poll so far and dropping any message that matches a seen mail ID. The trouble is that this is an O(N**2) operation that might significantly slow down the retrieval of large mail batches."

Virtual Mailboxes [AND MULTIPLE DELIVERED-TO, ARE THESE ISSUES ALWAYS INTERTWINED OR NOT?]

Virtual mailboxes which change the actual email address of the intended recipient whilst on the mailhost.

This probably results in none of the email being picked up by Fetchmail. This is a Fetchmail error message when dealing with this:
fetchmail: reading message catchall@example.org.uk:1 of 2 (1226 header octets) fetchmail: SMTP error: 550 <109-catchall@example.org.uk>: Recipient address rejected: User unknown in virtual mailbox table fetchmail: mail from FETCHMAIL-DAEMON@localhost.merci bounced to pete@sender.co.uk

There can be multiple 'Delivered-to:' instances in the header. Fetchmail might settle on one but it be the wrong one so you have to tell Fetchmail which is the correct one.

Trace the path from bottom to top through multiple 'Received:' fields. See at what point the address is there and at what point lost, for example:

Return-Path: <pete@sender.co.uk>
X-Original-To: richard@example.org.uk
Delivered-To: richard@example.org.uk
Received: from localhost (localhost [127.0.0.1])
     by file-server (Postfix)
Received: from localhost (localhost [127.0.0.1])
     by file-server (Postfix)
Delivered-To: 109-catchall@example.org.uk
Received: from example.org.uk [232.237.465.172]
     by localhost with IMAP (fetchmail-6.2.5)
Received: (qmail 23272 invoked by uid 110)
Delivered-To: 109-richard@example.org.uk
Received: (qmail 23269 invoked from network)
Received: from marley.nologic.org (82.195.144.138)
     by p15195002.pureserver.info with SMTP;
Received: from mail.nologic.org (localhost.localdomain [127.0.0.1])
     by marley.nologic.org (Postfix)
Received: from 82-52-18-63.dsl.in-addr.zen.co.uk ([82.52.18.63]) (proxying
     for unknown)
     (SquirrelMail authenticated user pete@sender.co.uk)
     by mail.nologic.org with HTTP;
To: richard@example.org.uk, alan@example.org.uk

In this instance you have the address in the lower 'Delivered-To:' and it's lost by the time of the upper. Fetchmail is processing the mail on the upper, the last one. Not only do we have to tell it to use 'envelope "Delivered-To:"' but we have to also tell it to instead use the lower 'Delivered-To:' field with 'envelope 1 "Delivered-To:"'.
Note that Receives are in different order on different hosts.
[THIS EXAMPLE MIXES THE VIRTUAL MAILBOX ("109-") PROBLEM WITH THE FACT THE CORRECT HEADER IS DELIVERED-TO]

We additionally have to tell Fetchmail to strip '109-' with the setting 'qvirtual 109-'.

Also possibly of use:

I saw advice saying to use 'Envelope-To' rather than 'Delivered-To' for qmail but there is no 'Envelope-To' field in this header.

Fetchmail multidrop configuration options

From the fetchmail info page

If fetchmail is used with a POP or an IMAP server, it has two fundamental modes of operation for each user account from which it retrieves mail: singledrop- and multidrop-mode. In singledrop-mode, fetchmail assumes that all messages in the user's account are intended for a single recipient. An individual mail message will not be inspected for recipient information, rather, the identity of the recipient will either default to the local user currently executing fetchmail, or else will need to be explicitly specified in the configuration file. Singledrop-mode is used when the fetchmailrc configuration contains at most a single local user specification for a given server account.

With multidrop-mode, fetchmail is not able to assume that there is only a single recipient, but rather that the mail server account actually contains mail intended for any number of different recipients. Therefore, fetchmail must attempt to deduce the proper "envelope recipient" from the mail headers of each message. In this mode of operation, fetchmail almost resembles an MTA, however it is important to note that neither the POP nor IMAP protocols were intended for use in this fashion, and hence envelope information is often not directly available. Instead, fetchmail must resort to a process of informed guess-work in an attempt to discover the true envelope recipient of a message, unless the ISP stores the envelope information in some header (not all do). Even if this information is present in the headers, the process can be error-prone and is dependent upon the specific mail server used for mail retrieval. Multidrop-mode is used when more than one local user is specified for a particular server account in the configuration file. Note that the forgoing discussion of singledrop- and multidrop-modes does not apply to the ESMTP ETRN or ODMR retrieval methods, since they are based upon the SMTP protocol which specifically provides the envelope recipient to fetchmail.

As each message is retrieved, fetchmail normally delivers it via SMTP to port 25 on the machine it is running on (localhost), just as though it were being passed in over a normal TCP/IP link. fetchmail provides the SMTP server with an envelope recipient derived in the manner described previously. The mail will then be delivered locally via your system's MDA (Mail Delivery Agent, usually sendmail(8) but your system may use a different one such as smail, mmdf, exim, postfix, or qmail). All the delivery-control mechanisms (such as .forward files) normally available through your system MDA a'nd local delivery agents will therefore work automatically.

--fetchdomains <hosts> (Keyword: fetchdomains) In ETRN or ODMR mode, this option specifies the list of domains the server should ship mail for once the connection is turned around. The default is the FQDN of the machine running fetchmail.

-Q <prefix> | --qvirtual <prefix> (Keyword: qvirtual; Multidrop only) The string prefix assigned to this option will be removed from the user name found in the header specified with the envelope option (before doing multidrop name mapping or localdomain checking, if either is applicable). This option is useful if you are using fetchmail to collect the mail for an entire domain and your ISP (or your mail redirection provider) is using qmail. One of the basic features of qmail is the 'Delivered-To:' message header. Whenever qmail delivers a message to a local mailbox it puts the username and hostname of the envelope recipient on this line. The major reason for this is to prevent mail loops. To set up qmail to batch mail for a disconnected site the ISP-mailhost will have normally put that site in its 'Virtualhosts' control file so it will add a prefix to all mail addresses for this site. This results in mail sent to 'username@userhost.userdom.dom.com' having a 'Delivered-To:' line of the form: Delivered-To: mbox-userstr-username@userhost.example.com The ISP can make the 'mbox-userstr-' prefix anything they choose but a string matching the user host name is likely. By using the option 'envelope Delivered-To:' you can make fetchmail reliably identify the original envelope recipient, but you have to strip the 'mbox-userstr-' prefix to deliver to the correct user. This is what this option is for.

-E <line> | --envelope <line> (Keyword: envelope; Multidrop only) In the configuration file, an enhanced syntax is used: envelope [<count>] <line>

This option changes the header fetchmail assumes will carry a copy of the mail's envelope address. Normally this is 'X-Envelope-To', but as this header is not standard, practice varies. See the discussion of multidrop address handling below. As a special case, 'envelope "Received"' enables parsing of sendmail-style Received lines. This is the default, and it should not be necessary unless you have globally disabled Received parsing with 'no envelope' in the .fetchmailrc file.

The optional count argument (only available in the configuration file) determines how many header lines of this kind are skipped. A count of 1 means: skip the first, take the second. A count of 2 means: skip the first and second, take the third, and so on.

Singledrop vs. Multidrop options

The 'is' or 'to' keywords associate the following local (client) name(s) (or server-name to client-name mappings separated by =) with the mailserver user name in the entry. If an is/to list has '*' as its last name, unrecognized names are simply passed through. Note that until fetchmail version 6.3.4 inclusively, these lists could only contain local parts of user names (fetchmail would only look at the part before the @ sign). fetchmail versions 6.3.5 and newer support full addresses on the left hand side of these mappings, and they take precedence over any 'localdomains', 'aka', 'via' or similar mappings.

A single local name can be used to support redirecting your mail when your username on the client machine is different from your name on the mailserver. When there is only a single local name, mail is forwarded to that local username regardless of the message's Received, To, Cc, and Bcc headers. In this case, fetchmail never does DNS lookups.

When there is more than one local name (or name mapping), fetchmail looks at the envelope header, if configured, and otherwise at the Received, To, Cc, and Bcc headers of retrieved mail (this is 'multidrop mode'). It looks for addresses with hostname parts that match your poll name or your 'via', 'aka' or 'localdomains' options, and usually also for hostname parts which DNS tells it are aliases of the mailserver. See the discussion of 'dns', 'checkalias', 'localdomains', and 'aka' for details on how matching addresses are handled.

If fetchmail cannot match any mailserver usernames or localdomain addresses, the mail will be bounced. Normally it will be bounced to the sender, but if the 'bouncemail' global option is off, the mail will go to the local postmaster instead. (see the 'postmaster' global option). See also BUGS.

The 'dns' option (normally on) controls the way addresses from multidrop mailboxes are checked. On, it enables logic to check each host address that does not match an 'aka' or 'localdomains' declaration by looking it up with DNS. When a mailserver username is recognized attached to a matching hostname part, its local mapping is added to the list of local recipients.

The 'checkalias' option (normally off) extends the lookups performed by the 'dns' keyword in multidrop mode, providing a way to cope with remote MTAs that identify themselves using their canonical name, while they're polled using an alias. When such a server is polled, checks to extract the envelope address fail, and fetchmail reverts to delivery using the To/Cc/Bcc headers (See below 'Header vs. Envelope addresses'). Specifying this option instructs fetchmail to retrieve all the IP addresses associated with both the poll name and the name used by the remote MTA and to do a comparison of the IP addresses. This comes in handy in situations where the remote server undergoes frequent canonical name changes, that would otherwise require modifications to the rcfile. 'checkalias' has no effect if 'no dns' is specified in the rcfile.

The 'aka' option is for use with multidrop mailboxes. It allows you to pre-declare a list of DNS aliases for a server. This is an optimization hack that allows you to trade space for speed. When fetchmail, while processing a multidrop mailbox, grovels through message headers looking for names of the mailserver, pre-declaring common ones can save it from having to do DNS lookups. Note: the names you give as arguments to 'aka' are matched as suffixes -- if you specify (say) 'aka netaxs.com', this will match not just a hostname netaxs.com, but any hostname that ends with '.netaxs.com'; such as (say) pop3.netaxs.com and mail.netaxs.com.

The 'localdomains' option allows you to declare a list of domains which fetchmail should consider local. When fetchmail is parsing address lines in multidrop modes, and a trailing segment of a host name matches a declared local domain, that address is passed through to the listener or MDA unaltered (local-name mappings are not applied).

If you are using 'localdomains', you may also need to specify 'no envelope', which disables fetchmail's normal attempt to deduce an envelope address from the Received line or X-Envelope-To header or whatever header has been previously set by 'envelope'. If you set 'no envelope' in the defaults entry it is possible to undo that in individual entries by using 'envelope <string>'. As a special case, 'envelope "Received"' restores the default parsing of Received lines.

Misc

If an address doesn't exist a message is supposed to be sent back to say that that mailbox doesn't exist. A consequence of multidrop is that all messages are delivered to the mailhost, then we collect them and don't send back an auto response that the mailbox doesn't exist. It isn't worth us sending back the auto response from our server because of SPAM / backscatter.

List mail puts the list address in the To: field

List mail puts the list address in the To: field, not the recipient address, so the email is lost because it's not addressed to anyone at the site. This is similar to the multiple recipients in the 'To:' field above.

Example 1 - 'X-Envelope-To'

To: refugee_support@yahoogroups.co.uk
X-Envelope-To: ourperson@example.org
Delivered-To: refugee_support@yahoogroups.co.uk

This is with a Zen Internet mailhost. 'To:' and 'Delivered-To:' contain a mailing list address 'refugee_support@yahoogroups.co.uk'. 'X-Envelope-To:' is where our intended email address is but Fetchmail isn't using it, instead it's using 'To:' and so people are receiving multiple emails. The Fetchmail documentation says it's most common in multidrop to use X Envelope To, which to me implies it would look for it but this doesn't seem to be the case. This required adding this to Fetchmail: 'envelope "X-Envelope-To"'.

Example 2 - 'Received'

To: csr-chicks@yahoogroups.com
Received: from n23.bullet.scd.yahoo.com ([66.94.237.52]) by mail.ourhost.co.uk (Kerio MailServer 6.3.1) for ourperson@example.org; Wed, 16 May 2007 01:29:11 +0100
Received: from [209.73.164.86] by n23.bullet.scd.yahoo.com with NNFMP; 16 May 2007 00:33:11 -0000
Received: from [66.218.67.103] by t8.bullet.scd.yahoo.com with NNFMP; 16 May 2007 00:29:11 -0000

'To:' contains the mailing list address. 'Delivered-To:' doesn't exist. The last 'Received:' contains the recipient, after the 'for'. So Fetchmail wants to use 'envelope "Received"'.

Gmail / Google Mail

This describes how to collect mail from a Google Apps for Domains or Gmail account using POP3.

Google enable POP3 collection by default.

Install package ca-certificates - Common CA Certificates PEM files. This is useful for any openssl applications to verify SSL connection. It includes the followings PEM files of CA certificates: spi-inc.org certificate, db.debian.org certificate, debconf.org certificate, Mozilla builtin CA certificates, CACert.org certificates, Brazilian Government Certificate, Signet CA certificates, QuoVadis CA certificates.

Fetchmail Configuration:

poll pop.gmail.com
	user "<Google mail username which is a full email address>" with pass "<Google password>" ssl sslcertck
		is <full mailbox address here> here keep

	user "<another Google mail username which is a full email address>" with pass "<Google password>" ssl sslcertck
		is <another full mailbox address here> here keep

'ssl' specifies to use ssl. 'sslcertck' specifies to check the certificates. Is 'sslcertpath' (specifies the path to the certificates) worth using?

In /var/log/syslog you'll see 'fetchmail: POP3< LOGIN-DELAY 300' which means Google want you to check for new mail no more often than every 300 seconds (5 mins)

Maildir Mailboxes (as per Christop Haas's 'Howto: ISP-style Email Server with Debian-Etch and Postfix 2.3')

Mail storage is dealt with by Dovecot rather than by Postfix. Dovecot uses Maildir and Maildir++ format mailboxes.

Maildir File And Directory Structure

This is the directory structure of the maildir mailbox:

Junk mail is in .Junk/cur and .Junk/new.

Deleted mail is in .Trash/cur, .Junk/new and .Junk/tmp.

From the man page on maildir:

"MAILDIR CONTENTS
A maildir contains three subdirectories: tmp, new, and cur. These three subdirectories comprise the primary folder, where new mail is delivered by the system. Folders are additional subdirectories in the maildir whose names begin with a period: such as .Drafts or .Sent. Each folder itself contains the same three subdirectories, tmp, new, and cur. Folders are not physically nested. A folder subdirectory, such as .Sent does not itself contain any subfolders. The main maildir contains a single, flat list of subfolders. These folders are logically nested, and periods serve to separate folder hierarchies. For example, .Sent.2002 is considered to be a subfolder called '2002' which is a subfolder of 'Sent'.
"

"MESSAGES
E-mail messages are stored in separate, individual files, one E-mail message per file. The tmp subdirectory temporarily stores E-mail messages that are in the process of being delivered to this maildir. The new subdirectory stores messages that have been delivered to this maildir, but have not yet been seen by any mail application. The cur subdirectory stores messages that have already been seen by mail applications.
"

Working With Maildir Files And Directories

See folders in a particular mailbox using ls -aCF /home/vmail/<domain>/<mailbox>/Maildir

Display the size of all Trash folders so you can look for excessive ones: find -iname Maildir -type d -exec du --max-depth=1 -h {} \;|grep .Trash

You can delete individual mail files at the command-line using the rm command (if the mail index is not OK, it's regenerated). This can be useful for example when there's a lot of junk mail, Thunderbird grinds to a halt when accessing the Junk folder / mailbox directory structure, or Thunderbird just takes a long time to delete when there's a lot of mail.

When there's an extreme amount of individual mail files to delete, they can be very slow to delete, or there be too many for rm to handle (giving "Argument list too long"). If you can safely delete all mail files in a particular directory then it's safe and vastly quicker to delete the directories 'cur', 'new' and 'tmp' themselves, using rm -rf <directory>, they'll then be recreated automatically when required.