FAQ: Using VM with procmail

0. This document

0.1 Why

Between January 1, 1998 and February 24, 1998, 17 messages about VM and procmail were posted to the gnu.emacs.vm.bug and gnu.emacs.vm.info newsgroups. Many of the questions asked had already been answered within the last 3-6 months. The VM FAQ doesn't currently contain a section on VM and procmail. I started to write an addition to the VM FAQ, but quickly realized that there's too much information on VM and procmail to quickly toss together. Thus, this document.

0.2 Where

This FAQ can be obtained from its canonical location, http://www.wonderworks.com/vm/vm-procmail.html. It is currently not being mirrored. It may be posted regularly to some newsgroups: gnu.emacs.vm.info comes to mind.

0.3 When

This document was last modified on: $Date: 2004/10/30 04:17:18 $
This is version (not revision!) 0.10 of this document.

0.4 Who

This FAQ is currently maintained by Samuel Mikes <smikes@alumni.hmc.edu>.

Several code examples are provided at the end of this FAQ; where I knew the author, I have attempted to give appropriate credit. If your code is mistakenly attributed to someone else -- or you'd like to contribute examples -- please contact the current FAQ maintainer.

I thank Brian Gorka and Era Eriksson for their helpful comments.

0.5 How

This FAQ is written in HTML using psgml-html mode under XEmacs 20.3. The syntax of the document is verified against the HTML 3.2 Final DTD.

Text versions of this FAQ (such as those posted to newsgroups) are created with lynx (lynx -dump -nolist).

0.6 Future

Things that I'd like to do with this document:

1. Locking and spool files

1.1 "I'm using procmail to save messages to a folder..."

This is the most common problem. You should not use procmail to save messages to a VM folder. You should use procmail to save messages to a spool file, and let VM pick up the mail from the spool file.

In your VM configuration file (~/.vm or ~/.emacs):


;;; first, set up special handling for the system mailbox
(setq vm-spool-files (list
                       ;; this should be a list of triples:
                       ;;    <mail folder>       <spool file>    <crash box>
                       (list "~/priv/mail/inbox" (getenv "MAIL") "~/priv/mail/inbox.crash")))

;; now set the defaults for other folders:
(setq vm-crash-box-suffix ".crash")
(setq vm-spool-file-suffixes (list ".spool"))

This tells VM to look for system mail in the location specified by the environment variable $MAIL, put it in ~/priv/mail/inbox, and use inbox.crash as a crash box to protect you from catastrophic loss of mail. You will want to modify this sample code from my setup, since most people don't keep their mail in ~/priv/mail/inbox.

For all other folders, VM will look for a file named "<file>.spool" and incorporate mail from that.

Your procmailrc should then deliver mail into the spool files:


# .procmailrc

:0:
* ^Subject:.*\[foo-users\]
foo-users.spool

:0:
* ^Sender:.*baz-request
baz-request.spool


Now, if you visit the folder "baz-request", vm will automatically pick up mail from "baz-request.spool" and incorporate it into the folder. If new mail arrives for 'baz-request" while you're visiting the folder, you can bring it in with vm-get-new-mail, normally bound to "g".

If the variable "vm-spooled-mail-interval" is set, then VM will periodically check for new mail: if a visited folder has new mail waiting, the "Mail" indicator will be displayed in its buffer's mode-line. If the variable "vm-auto-get-new-mail" is set to a number (n), VM will automatically retrieve new mail for visited folders, polling every n seconds.

Some other ways of doing this are presented below.

1.2 "Why do I have to use spool files?"

The problem stems from the fact that emacs (and thus VM) have a different philosophy of file locking from sendmail, procmail, and the like.

The procmail (and /bin/mail, and sendmail, etc..) locking model is designed for a system where many programs (or many instances of the same program) wish to modify the same file. Each program wants to lock the file, append the message it's responsible for, and unlock the file. This locking model works well for mail delivery.

The emacs locking model is designed to allow lots of people to look at files without modifying them. emacs doesn't try to lock a file until you start to modify it. This lets a number of people view the same file by holding a buffer in memory unmodified and unlocked. The first person who tries to modify (not open!) the file gets the lock.

VM expects that no-one else will modify its mail folders while it's working with them. When you visit a mail folder, VM reads the folder from disk into a buffer, and manipulates the buffer in memory (marking, deleting, and saving messages, for example). When you tell VM to incorporate new mail into the folder (vm-get-new-mail, by default bound to "g") or save the folder, VM checks to make sure the file on disk hasn't changed since it was read. If it has, VM prints the dreaded "Folder inbox changed on disk, consider M-x revert-buffer" message.

To me, the cleanest solution is to not use procmail to deliver to VM's folders. However, dissenting opinions exist; see below.

1.3 "What's this in the procmailex(5) man page about emacs lock-files?"

The procmailex(5) man page says (in part)

       When delivering to emacs folders (i.e. mailfolders managed
       by any emacs mail package, e.g. RMAIL or VM) directly, you
       should use emacs-compatible lockfiles.  The emacs  mailers
       are  a  bit  braindamaged  in  that respect, they get very
       upset  if  someone  delivers  to  mailfolders  which  they
       already  have  in  their  internal buffers.  The following
       recipe assumes that $HOME equals /home/john.

              MAILDIR=Mail

              :0:/usr/local/lib/emacs/lock/!home!john!Mail!mailbox
              * ^Subject:.*whatever
              mailbox

       Alternatively, you can have procmail deliver into its  own
       set  of  mailboxes,  which you then periodically empty and
       copy over to your emacs files  using  movemail.   Movemail
       uses mailbox.lock local lockfiles per mailbox.

The first paragraph and the accompanying example talks about delivering mail to emacs folders, which is a bad idea, as we've discussed above.

The second paragraph talks about using procmail to deliver to "its own set of mailboxes" -- i.e., spool files -- and moving the mail using movemail. That's what VM does with the mailbox-spoolfile-crashbox system.

Since movemail uses the procmail "<file>.lock" locking convention, you should not configure procmail to use emacs-style locking for your spool files. Using the normal ":0:"-style locking is enough.

1.4 "What's another way to do this?"

Everybody has their own way of doing this, of course; that's the beauty of VM.

Sample code is provided at the end of this document, in the resources section under sample configurations

2. POP

2.1 "Can I use VM's POP capability together with procmail?"

Nope.

But you can use any of a number popmail clients, and configure them to use procmail as a local delivery agent. I recommend Eric S. Raymond's fetchmail, available at http://www.ccil.org/~esr/fetchmail. fetchmail works best with a local mail transfer agent such as sendmail, but it can be configured to directly use a local mail delivery agent, such as procmail.

2.2 "How do I go about using fetchmail to deliver to my MTA?"

(I assume that you have a mail transfer agent (MTA) such as sendmail or qmail installed, and that you're using procmail for local delivery -- either as the default mail delivery agent (MDA), or through a forward file as described in the procmail(1) man page. If you don't have an MTA installed, either install one or skip to the appropriate section of this FAQ.)

Obtain and install fetchmail and follow the directions for setting up a .fetchmailrc. For a simple case, your .fetchmailrc will look like mine:

# .fetchmailrc
poll pop.teokem.lu.se
user teosom there is smikes here

Run fetchmail, either interactively ($ fetchmail) or as a daemon ($ fetchmail -d 30).

If you don't have a mail transfer agent installed and listening on port 25 (smtp), this won't work. So, hie yourself over to http://www.sendmail.org/, or try the next option.

2.3 "How do I go about using fetchmail to deliver to my MDA (procmail)?"

Obtain and install fetchmail (as before).

Create a .fetchmailrc which tells fetchmail to hand off your mail to procmail for delivery (we'll use formail to preprocess the mail and hand it off to procmail)

# .fetchmailrc
poll pop.teokem.lu.se
user teosom there is smikes here
and wants mda "/usr/bin/formail -ds procmail"

WARNING: YOU MAY LOSE MAIL.. The author of fetchmail, Eric S. Raymond, says this about the "mda" option of fetchmail (from the fetchmail manual):


You can force mail to be passed to an MDA  directly
(rather than forwarded to port 25) with the -mda or
-m option.  If fetchmail is  running  as  root,  it
sets  its  userid  to that of the target user while
delivering mail through an MDA.  Some possible MDAs
are  "/usr/sbin/sendmail  -oem", "/usr/lib/sendmail
-oem", "/usr/bin/formail", and  "/usr/bin/deliver".
Local  delivery addresses will be inserted into the
MDA command wherever you place a %s.  Do not use an
MDA  like "sendmail -oem -t" that dispatches on the
contents of To/Cc/Bcc, it will  create  mail  loops
and  bring  the just wrath of many postmasters down
upon your head.

...

                            ... When forwarding to an MDA,
however, there  is  more  possibility  of  error  (because
there's  no  way  for fetchmail to get a reliable positive
acknowledgement from the MDA).

Since direct delivery to an MDA can lose mail, I recommend installing and using an MTA, such as sendmail. An added benefit of installing sendmail is that you can use the check_mail anti-spam rules maintained by Claus Assmann.

3. Notification of Incoming Mail

3.1 VM

You can make VM automatically check for and incorporate mail from spool files. Consider the following variables:


`vm-auto-get-new-mail' (buffer: *Hyper Apropos*, mode: Hyper-Apropos)

User variable:

  value: nil

  *Non-nil value causes VM to automatically move mail from spool files
  to a mail folder when the folder is first visited.  Nil means
  you must always use vm-get-new-mail to pull in newly arrived messages.

  If the value is a number, then it specifies how often (in
  seconds) VM should check for new mail and try to retrieve it.
  This is done asynchronously and may occur while you are editing
  other files.  It should not disturb your editing, except perhaps
  for a pause while the check is being done.

---

`vm-mail-check-interval' is a variable declared in Lisp.

Value: nil

Documentation:
*Numeric value specifies the number of seconds between checks
for new mail.  The maildrops for all visited folders are checked.

A nil value means don't check for new mail.

Note that mail if new mail is found, it is not retrieved.  The
buffer local variable vm-spooled-mail-waiting is set non-nil in
the buffers of those folders that have mail waiting.  VM uses
the displays "Mail" in the mode line of folders that have mail
waiting.

If you set vm-auto-get-new-mail to `t' and vm-mail-check-interval to some number, then new mail will be incorporated into a buffer when you first visit it. While you're visiting it, if new mail arrives, the "Mail" indicator will appear in your mode-line, and you can hit "g" to get the new mail.

On the other hand, you could set vm-auto-get-new-mail to, say 10 -- in which case, new mail would be incorporated into visited buffers every ten seconds.

These variable definitions are from VM 6.43. Use `describe-variable' and `apropos' on your version of VM to see whether it has these variables. If you're stuck with an older version of VM (you run an older version of FSF emacs, say), you should consider one of the alternative options.

3.2 xbuffy

xbuffy is a generalization of xbiff. It can watch multiple spool files, execute a user-defined command when mail arrives (by default, it beeps), display a summary of the incoming mail, and execute a user-defined command when you middle-click on a mailbox. (This could be used, for example, to run gnuclient(1) and tell your emacs to visit the folder.)

xbuffy can be obtained from an X11 archive site, such as ftp://ftp.sunet.se/pub/X11/contrib/utilities/. The most recent version of xbuffy as of the date printed on this FAQ is 3.3

3.3 mspools.el

There is reputed to be some sort of multiple-spool watching code written in elisp by Stephen Eglen. I haven't tried it.

The code is available through Stephen's emacs page, or directly at http://www.cns.ed.ac.uk/people/stephen/emacs/mspools.el

3.4 comsat

Since procmail does the normal comsat-notification when new mail arrives, you could build some sort of mail-notification around the comsat(8) daemon. Just a thought, though: I don't think this is a terribly good idea.

3.5 A procmail log monitor

(Suggested by era eriksson <era@iki.fi>)

Since procmail can be configured to log all received mail (by specifying a logfile in the .procmailrc), it would be possible to cook up your own mail-monitor by watching the procmail log file for deliveries.

4. Other Resources

4.1 VM Resources

The latest version of VM can be obtained from ftp://ftp.uu.net/in the directory networking/mail/vm/ as vm.tar.gz.

There are two VM newsgroups: gnu.emacs.vm.info for general questions and discussion, and gnu.emacs.vm.bug for bug reports. (Bug reports should be submitted with M-x vm-submit-bug-report.) The newsgroups are also gatewayed to mailing lists, as described in the MAILINGLISTS file in the emacs distribution.

The VM FAQ (http://www.wonderworks.com/vm/FAQ.html) was written by Brian Gorka. It answers general questions about VM, its user interface, and configuration.

A library of VM code can be found at Emmett Hogan's VM Add-Ons web page, http://www.gnac.com/~hogan/vm/. He has several sample configurations and some neat hacks and enhancements, including the spookmime.el MIME boundary generator.

4.2 procmail resources

procmail can be obtained from many locations; the primary source is ftp://ftp.informatik.rwth-aachen.de/pub/packages/procmail/.

A Procmail FAQ is at http://www.iki.fi/~era/procmail/mini-faq.html. The maintainer of that FAQ also maintains a list of procmail links, which includes pointers to example recipes and mailing lists.

4.3 Mail Filtering resources

The Infinite Ink Mail Filtering FAQ can be obtained through their faq-launcher at http://www.ii.com/internet/faqs/launchers/mail/filtering-faq/

4.4 Example configurations

Neal Young <Neal.Young@dartmouth.edu>

;; run vm with file-completion on folders with waiting mail

;;;###autoload
(defun ney-vm ()
  "vm primary mbox.  Prompt for folder, defaulting to first with mail waiting,
and completing on non-empty.  with prefix-arg, prompt for file name instead"
  (interactive)
  (require 'vm)
  (vm-session-initialization)
  (let ((this-command 'vm-visit-folder)
        (folder
         (if current-prefix-arg
             (read-file-name "Run vm on file: " nil nil t)
           (let* ((waiting (ney-vm-folders-with-mail-waiting))
                  (default (car waiting)))
             (completing-read "Run vm on folder: " (mapcar 'list waiting)
                                nil nil
                              (and default (cons default 0)))))))
    (vm-visit-folder (or folder vm-primary-inbox))))

(defun ney-vm-mail-waiting-p (inbox)
  (let ((attributes (file-attributes
                 (file-truename (expand-file-name inbox)))))
    (and
     attributes                         ; file exists
     (not (eq (car attributes) t))      ; not directory
     (> (nth 7 attributes) 0))))        ; not empty

(defun ney-vm-folders-with-mail-waiting ()
  (ney-reset-vm-spool-files)            ; defined below
  (delete nil
          (mapcar
           (function
            (lambda (triple)
              (if (ney-vm-mail-waiting-p (nth 1 triple))
                  (file-name-nondirectory (nth 0 triple)))))
           vm-spool-files)))

;; set "vm-spool-files" automatically according to existing spool files
;; I define and run this in my .vm file and in procedure ney-vm below.
;; assumes spool files will exist even if empty

(defun ney-reset-vm-spool-files ()
  (setq vm-spool-files
        (append
         (list
          (list (my-folder "in")
                (or (getenv "mail") "/usr/spool/mail/ney")
                (my-crashbox "in")))
         (mapcar '(lambda (x)
                    (list (my-folder x) (my-inbox x) (my-crashbox x)))
                 (select-list-items "^[^.]"
                                    (directory-files
                                     (concat vm-folder-directory
                                             "spool")))
                 ))))

kerryt@akl.optimation.co.nz (Kerry Thompson)

.forward:
    "|IFS=' '&& exec /usr/bin/procmail -m /home/kerryt/.procmailrc ||exit 75"
# .procmailrc
PATH=/bin:/usr/bin:/usr/bin
MAILDIR=$HOME/Mail      #you'd better make sure it exists
#DEFAULT=$MAILDIR/mbox   #completely optional
LOGFILE=$MAILDIR/procmail.log   #recommended

:0:
* ^TOdjb-qmail
qmail-in

:0:
* ^TOexim-users
exim-in

:0:
* ^TOinfo-vm
vm-in

#:0
#* ^Subject:.*flame
#/dev/null

This tells procmail to stick matching (VM list) messages into ~/Mail/vm-in, all of my mail folders live in ~/Mail. Then I setup vm-spool files in my ~/.vm :

(setq vm-spool-files
      '(("~/INBOX" "/var/spool/mail/kerryt" "~/Mail/INBOX.CRASH")
        ("qmail" "~/Mail/qmail-in" "~/Mail/qmail.CRASH")
        ("exim" "~/Mail/exim-in" "~/Mail/exim.CRASH")
        ("vm" "~/Mail/vm-in" "~/Mail/vm.CRASH")
))

Gael Marziou <Gael_Marziou@hp.com>

;; to work with mailagent and xbuffy
;; mailagent is a mail filter like procmail
;; filtered mail is put into spool files located under ~/var/mail/
;; the functions below tell vm how to retrieve the spool file name
;; associated to a VM folder.
(defconst my-mail-spool-dir "~/var/mail/")
(defun my-vm-make-spool-file-name (folder)
  (let*
      ((my-vm-folder-directory (expand-file-name vm-folder-directory))
       (indx (string-match my-vm-folder-directory (expand-file-name
      folder))))
    (if indx
        (concat my-mail-spool-dir
                (substring (expand-file-name folder)
                           (+ indx (length my-vm-folder-directory))))
      folder)))
(setq vm-make-spool-file-name 'my-vm-make-spool-file-name)

Raymond Wiker <etorwi@eto.ericsson.se>

... A better way of doing this is to send the mail to am different file, and tell vm to fetch mail for INBOX from this file. In my setup, I use

(setq vm-spool-files
      (list
       (mapcar 'expand-file-name
               '("~/INBOX" "~/INBOX.CRASH" "~/Mail/new"))
       (mapcar 'expand-file-name
               '("~/INBOX" "~/INBOX.CRASH" "/var/mail/etorwi"))
       (mapcar 'expand-file-name
               '("~/Mail/spam" "~/Mail/spam.crash" "~/Mail/spam-p"))
       (mapcar 'expand-file-name
               '("~/Mail/spam" "~/Mail/spam.crash" "~/Mail/spam-new"))))

David Maslen <david@binary.net.au>

David posted this code to gnu.emacs.vm.info, but he didn't write it, so please don't go bothering him about it! If the author of this code would like to take responsibility for it, please contact the FAQ maintainer

;; Since I use procmail to filter my mail, I like to shove my incoming
;; mail to folders. Due to Emacs (and hence VM) locking, it's not wise
;; to view the incoming spool file, so I make procmail dump mail to
;; folders IN ALL CAPS. Then I have VM make spool files from each one of
;; those, except they'll be in all lower case.

(setq vm-spool-files
      (append
       (list
        (list vm-primary-inbox
              "~/Mailbox"
              (concat vm-primary-inbox ".crash")))
       ;; Mailing list inboxes
       ;; I arrange that all my procmail maildrops are in ~/Mail/[A-Z]*
       (mapcar '(lambda (s)
                  "make the appropriate entry for vm-spool-files"
                  (list
                   (concat vm-folder-directory "lists/" s)
                   (concat vm-folder-directory "lists/" (upcase s))
                   (concat vm-folder-directory "lists/" s ".crash")))
               ;; So I create a vm-spool-files entry for each mail drop
               (mapcar 'downcase
                       (directory-files
                        (concat vm-folder-directory "lists/")
                        nil "^[A-Z].+")))))

"J. Daniel Smith" <DanS@bristol.com>

From a post to gnu.emacs.vm.info: edited and HTML-ized by Sam Mikes

well, I finally figured out today that Emacs 19 was writing its lock files in a different directory! I'm not sure I completely understand your reasons for wanting VM and procmail to have different files (a "spool file" and a "folder"), but I will concede I'm "proceeding at my own risk".

The original thread was the frequent mixing of procmail and VM. The usual solution for things like mailing lists is to have ~/.procmailrc write things to a "spool file", and then tell VM in ~/.vm about the relationship. My complaint about this scheme is that it requires me to keep the two files in sync, something I'd rather not do.

Other items were dealing with how I manage my mail with "procmail" - most of the stuff I get is of a "read and delete" nature; thus I have procmail sort incoming mail into files in a "toread" directory. Then, in VM, I can visit the "toread" directory and using Emacs' file completion to easily see what new mail I have.

The various solutions with vm-spool-file-suffixes or putting spool files in a different folder all have various problems with this scheme:

I *think* everything would be solved to my liking with the following two enhancements to VM

Anyway, my solution for now is to attack the problem from the other direction - that is, make procmail use Emacs' locks. That way when I'm visiting a folder in VM, procmail won't try to dump anything into it.


Last updated on Apr 23 1998 at 22:26 by Samuel Mikes