Effectively supporting virtual domains with Mailman

When I first decided to run my own list server, I was well aware of the popularity of Mailman but I was deterred from using it by the complexity of setting up virtual hosts. In particular, 99% of the software I use for servers is packaged. Whenever I choose a new piece of software, using something that is packaged and supported is high on my list of priorities. Mailman's distribution tarball and the packages built from it only have limited support for virtual hosting.

My own objective is to host lists for open source projects. As some of these projects don't have dedicated budgets, it is particularly important to find lightweight, low-maintenance solutions. Furthermore, it is important that those solutions don't detract from the time I want to invest in the projects themselves.

I've developed some scripts to solve the problem: to start using them quickly, please see the README file. Here I provide a more detailed overview of the logic behind this approach.

Mailman's limitations and workarounds

For me, the biggest limitation of Mailman is that the site list is always linked to a single domain, which has to be chosen arbitrarily from the list of domains being hosted. Although it is not emphasized in the documentation, there is also the inconvenience of having all the lists from potentially unrelated projects appearing together on Mailman's main index page. Another limitation is that each list name must be globally unique across all domains.

Some existing workarounds exist but they have limitations too. There is a virtual domain patch, but it is not clear how this handles the site list or the site index page. There is also the postfix-to-mailman script distributed by Debian and Ubuntu: using a transport-based solution looks elegant at first, but when you realise that you need to whitelist the addresses (to ensure messages are rejected at ingress rather than bounced), it becomes just as much work to maintain as the alternatives it aims to avoid: Mailman's authors do not endorse this approach.

Experimenting with Sympa

Deterred by Mailman's limitations, I initially went with Sympa and that seemed to run quite well for a number of years. In particular, it supports full virtual hosting out-of-the-box and all this is standard in the convenient Debian packages.

From time-to-time, users have complained about the use of JavaScript in the web UI, but that was not something I could immediately resolve.

However, when I upgraded to Debian 7 (wheezy), more serious problems appeared.

Initially, the web UI (used for subscribing or browsing the archives) was completely broken. Anybody accessing the site would be given a download of the wws binary rather than the page that was supposed to be generated by the binary. I discovered errors about this in Apache's error.log and tried installing libapache2-mod-fcgid. The problem was resolved, but from then on, my whole web server became unstable and at least once every day I would have to restart the Apache process or it would become inaccessible. I put the details into a bug report and after not getting any feedback, decided to go and have another look at Mailman.

A closer look at Mailman

Looking at Mailman again, it appeared that the only way to achieve a relatively "pure" virtual hosting solution would be to run multiple installations of Mailman in different directory trees.

Initially, I hoped this could be done by sharing copies of the binaries and just creating replicas of the structure under /var/lib/mailman. Looking under the hood, I discovered that the paths were compiled into the binaries and could not be easily overriden. It appears that this limitation could be overcome with some trivial patches to read directory names from the environment.

The main aims of using shared binaries are disk space efficiency and also being able to benefit from security updates for the binary packages. However, if I had to patch the binaries, then I wouldn't be able to benefit from the security update packages anyway.

So, it became inevitable that I would have to build binaries from source one way or another: either with a patch or by building multiple copies of each binary, each hardcoded with a different path.

Putting it into practice

Looking over the Mailman architecture, there are four things to be replicated for each installation:

  • init script to start a standalone qrunner per domain
  • cron jobs
  • CGI configuration from Apache (easily isolated by using a different ScriptAlias inside each Apache <VirtualHost>
  • Invocation of the Mailman binary from the mailer's aliases file

The init scripts and crontabs can be easily generated from a template (I used Debian's copies of these files as a base for that). Some of the Apache configuration can be handled by putting wildcards in the <Directory> specifications and the rest can be easily cut-and-pasted into <VirtualHost> definitions.

Based on all of the above, all I had to do was run Mailman's configure script and make once for each virtual domain. I've scripted the whole process - my script is available here

Managing the aliases file

One particular limitation of my mailer (Postfix) is that the Mailman message processor can't be invoked directly from a rule in the virtual alias map. Scripts like this can only be invoked from the normal /etc/aliases file. This problem is not unique to Mailman: for Sympa, I also had to deal with this. The most common solution is to use the virtual file to translate all the virtual list addresses into very verbose usernames within the local domain:

Address Mapped to
widget-users@lists.example.org lists.example.org-widget-users@my-local-domain

The local part, lists.example.org-widget-users can then be used in the regular /etc/aliases file.

This approach is described in the Sympa documentation and also in one of the virtual domain patches for Mailman.

I've included a script for generating these alias mappings in the repository with my other scripts for building virtual Mailman installations. It would also be possible to patch Mailman itself to generate such files dynamically when new lists are created.

Conclusion and next steps

Using the techniques described and these small scripts, I successfully built independent virtual hosts for two of my domains, lists.lumicall.org and lists.dynalogin.org on a single mail host. Notice how each of them has the appearance of a clean, standalone Mailman installation?

Going forward, I believe it would not be hard for the 2.x branch of Mailman to support this approach to virtual hosting with three changes:

  • Allowing the binaries to take their directory tree name from an environment variable or command line option or by inspecting argv[0] to discover their own location,
  • Generating the aliases file in the more verbose format and
  • Providing some convenient init script and crontab that iterates over all the domains

Comments

Virtual hosting limitations and things like the site list are supposedly solved in Mailman 3, which is currently labeled "beta". However, there have been no Debian packaging efforts just yet.

I'm using mailman in wheezy (i.e. 1:2.1.15-1) and it seems to support virtual domains quite well following the instructions in README.Exim4.Debian.gz and using the default setup for apache.

As far as I can tell, what you describe will work, but it will still have the limitations of the "vanilla" Mailman's support for virtual domains, e.g. all the lists from all domains appearing in the master web index, or a site-list tied to a specific domain.