The Exim-Python Extension

Exim

Exim is a message transfer agent (MTA) developed at the University of Cambridge for use on Unix systems connected to the Internet. There is a great deal of flexibility in the way mail can be routed, and there are extensive facilities for checking incoming mail.

http://www.exim.org/

Python

Python is an interpreted, interactive, object-oriented programming language. It is often compared to Tcl, Perl, Scheme or Java. Python combines remarkable power with very clear syntax. It has modules, classes, exceptions, very high level dynamic data types, and dynamic typing. There are interfaces to many system calls and libraries.

http://www.python.org/

Exim-Python

Exim-Python is the name for an extension to Exim which adds the ability to execute compiled Python functions and methods from within Exim configuration file string expansions. Since Exim uses string expansions heavily throughout it's operation, this feature allows powerful added control to an already very flexible mail server.

http://botanicus.net/dw/

Exim alone is an extremely powerful mail server, approaching as feature complete as one can expect from any mail server software. It is also fairly easy to extend Exim using native code, but to experiment with new feature implementation like this takes a lot of time.

This extension allows you to prototype a new feature for Exim quickly and easily. To say it is useful only as a prototyping tool would be incorrect, indeed the performance of the extension and the code running beneath it should suffice for all but the highest volume mail servers.

Other potential uses of the extension include data source abstraction, for example, using the Python DB-API, you can make your Exim installation database-independant by calling a layer of functions that perform lookups using DB-API compatible interfaces rather than directly themselves.

If you have questions regarding the usage or behaviour of this extension, please contact the author at the address at the top of this document.

Prerequisites

You must be using a version of Python that supports the boolean protocol. This means that a minimum version of Python 2.2.1 is required for operation of the extension.

The extension is distributed in two versions - a patch and a pre-patched tarball. It is recommended that you use the pre-patched tarball unless you need to further extend Exim with other third party patches.

Note: the pre-patched tarball also includes the latest version of the exiscan-acl patch, available from http://duncanthrax.net/exiscan-acl/.

This has little impact on the operation of Exim if you don't use it's features, however it is a very common requirement in modern Exim installations, and as such, is included for ease of installation. The usefulness of this extension is also greatly increased by the presence of the extra ACLs.

Manually patching the Exim distribution.

Download and extract the latest Exim tarball from http://www.exim.org/, and the accompanying patch from http://botanicus.net/dw/:

$ cd /usr/src
$ wget http://www.exim.org/ftp/exim4/exim-4.32.tar.bz2
$ wget http://botanicus.net/exim-python/exim-4.32py1.patch
$ tar jxvf exim-4.32.tar.bz2

Enter the Exim source directory, and apply the patch:

$ cd exim-4.32
$ patch -p1 < ../exim-4.32py1.patch

Configuring the Exim sources.

Follow the Exim installation procedure outlined in the specification: http://www.exim.org/exim-html-4.30/doc/html/spec_4.html.

In Local/Makefile, you will additionally need to specify the EXIM_PYTHON, PYTHON_CC, PYTHON_CCOPTS, and PYTHON_LIBS variables. The defaults should suffice, although you may need to adjust the included paths if you have installed Python to a non-standard location.

Additional configuration directives.

In order to enable the extension, you must tell it the name of a Python source file to load and compile at startup. Any modules, packages, functions, and classes that you wish to use should be defined in this file, or imported into it using the Python import statement.

python_at_start

The boolean python_at_start parameter indicates to Exim whether or not it should load the extension at startup. It defaults to false, to help reduce daemon memory usage slightly. Setting it to true will increase the performance of the server when Python calls are made later on. This is recommended for production servers.

python_startup

The string python_startup parameter specifies the name of the Python source file to load and compile at startup. This parameter must be set to a valid (existing, and syntactically valid) source file pathname in order for the extension to be used.

Example:

# Parse and load a Python module for use in ${python} expansions.
python_at_start = true
python_startup = /etc/exim/exim.py

How the extension loads and runs your code.

File loading.

At startup, the extension will attempt to read in the contents of the file specified by the python_startup configuration parameter, compile it, and execute it as a module.

Usage of "__name__".

When the extension executes your code, it sets the module-level variable __name__ to __exim__. The usage semantics of __name__ are aligned with how Python uses it.

By testing __name__, it is possible (although impractical in many cases) to write a single Python source file that will operate as a standard Python module, an Exim-Python module, and as a command-line tool. Here is an example:

if __name__ == '__main__':
    run_commandline()

elif __name__ == '__exim__':
    init_exim_module()

else:
    pass # Running as a Python module, do nothing.

How Python functions and methods are called.

When the extension encounters a ${python {<fn>} ...} expansion, it resolves the name <fn>. In the module loaded previously. ... may be one or more string arguments that will be passed to the Python function, encapsulated in braces.

Return values and exceptions.

Several rules are applied to the result of your Python function, and it is very important you understand them. Note: it is an error if your Python function does not return a value!

• If the called function returns None (implicit default if no return with value executed), then the expansion is forced to fail.
• If the called function generates an exception, the exception will be printed on sys.stderr, a log will be made in the paniclog, and the expansion will be forced to fail.
• If the called function returns True or False (the boolean singletons), these are converted to the strings true and false respectively.

Loading and initialising module resources.

Exim is a rather complex piece of software, and the execution path it takes can be quite confusing at times. For this reason, it is desirable to be as careful as possible when initialising and using resources.

The problem.

Exim forks quite regularly, and much of it's operation happens inside a subprocess. When Exim performs a fork, any objects existing in the Python interpreter at the time of fork will now be duplicated.

This can lead to dangerous and incorrect use of, for example, database connections, which will now have essentially two clients connected to only one client's connection.

The extension as yet does not provide any automatic means for handling objects that should be destroyed or reinitialised at fork time, so it is up to you to do this.

A simple solution.

Here, our dangerous resource is created and destroyed every time the function is entered. This avoids the problem in almost all cases.

def get_db():
    db = MySQL.connect(**mysql_connection_details)
    return db, db.cursor()


def lookup_password(username):
    db, cursor = get_db()

    if not cursor.execute("""SELECT ..."""):
        return None # force failure.

    return cursor.fetchone()[0]

Restrictions

Every feature of the Python environment is not available from within Exim. Certain limitations are placed on your code to avoid destablizing the Exim server process. Here they are:

Avoid the use of threads.

Or, do so at your own risk. There are many complications associated with the use of threads, and you can almost certainly accomplish your goal without using them.

Avoid signal manipulation.

At this time, it is currently unknown how Exim would deal with restarting system calls in the case of a signal delivery handled by the extension. It is recommended that you avoid all use of the signal module until usage documentation appears in a future release of this extension.

Other usage notes.

• If your code opens any files, it is recommended that you use the fcntl.FD_CLOEXEC flag on them, in order to enhance security, and keep the files within their required scope.
• If you are seeing failed string expansions, but no further traceback information is being provided, you may want to reopen the sys.stderr object in your module, directed to a file in /tmp. A better solution to this is in the works.
• If you instantiate a class, or wish to write directly to a function or method or callable class deep inside a module hierarchy, you may do so. Example: # Python code:
  import sys
  
  # Exim expansion:
  ${python {sys.stderr.write} {$sender_host_address}}

Deny connections from certain countries using GeoIP.

This example combines the extension with Exim's built-in access control lists. Here, we prevent connections to the mail server unless they originate a country that is specifically authorised to send e-mail to us. This is an effective, if dangerous, anti-spam measure for English speakers.

It requires the GeoIP client library, a copy of the GeoIP database, and the GeoIP Python bindings. See http://www.maxmind.com/app/python.

Python Module

import GeoIP

gi = GeoIP.new(GeoIP.GEOIP_MEMORY_CACHE)
allowed_countries = [ 'UK', 'US', 'CA', 'EU', 'IE' ]


def connect_denied(remote_addr)
    if not remote_addr: # Local connection.
        return False # Allow it.

    country = gi.country_code_by_addr(remote_addr)
    return country not in allowed_countries

Exim Configuration

# My "acl_smtp_connect" ACL:

acl_check_connect:
    deny condition = ${python {connect_denied} {$sender_host_address}}
         message   = "Your country is not allowed to connect to this server."

January 21st, 2006.

• Merged the patch with exim-4.60.

April 25th, 2004.

• First public beta release.
• Added HTML documentation.
• Added obj.subobj.subobj.callable resolution.
• Fixed silly reference counting bug in python_resolve().

February 20th, 2004.

• Initial public alpha release.

Needs fixed.

• When running Exim in debug mode, for example, with exim -d -bd, pressing the interrupt key will not shut Exim down. This is evidence that Python is still further damaging signals or some other behaviour, and requires investigation.

Needs tested.

• Verify that save_signals() and load_signals() in python.c are working effectively.
• Ensure that PyOS_AfterFork() is called only when necessary.
• Check the safety of while () predicates in python.c:python_resolve().

Needs implemented.

• Add memory checking to python.c:call_python_cat() in the arg tuple build.
• Add memory checking to python.c:save_signals().
• Remove the requirement for #undef HAVE_SETEUID in python.c. Find out if it is Exim or Python being ignorant.
• Add support for ${pythonkw {keywordlist} [{arg1} [{argn} ..]]} for calling objects that can accept keyword arguments. keywordlist will be of the exim form key=val key2=val2 and so on.
• Add support for converting Python dictionaries into Exim-style key=val strings on return from a function.
• Make Python traceback output configurable to syslog, file, standard error, or Exim's paniclog.

The wishlist.

• Add a standard way of registering objects that need reinitialised after a fork. Some sort of (<parent>, <object_name>, <constructor>, <constructor_args>, <constructor_kwargs>, <reinit_where>) 6-tuple?
• Add support for modifying certain Exim variables at run-time, for hot reconfiguration, and so fourth. This would require the ability to safely trigger Python code at arbitrary locations in Exim. A mammoth task, and a potentially involved implementation.
• Integrate the exim-python.html documentation file into the Exim specification, and provide the patched documentation online as HTML.
• Make available a python router, transport, authenticator, and lookup. This might be useful to some, for example, for SQL-backed e-mail storage.
• Add a general exim module, to allow certain internal Exim functions be called from Python. This would also encapsulate the run-time reconfiguration wishlist item.