User Tools

Site Tools


programming:python:python-libmilter

This is an old revision of the document!


python-libmilter

About

python-libmilter started as a project to maximize the portability of libmilter. I know about, and have used, the excellent CPython implementation by Stuart D. Gathman py-milter. Though I liked this library for it's thin wrapper around the C libmilter library (and thereby speed), I had some issues with getting it to install, having to find and download the appropriate source code for libmilter version installed, etc. This is why I went with a ground up implementation of libmilter in pure Python. I want something I could just “drop in” to a project and have a milter up and running in almost no time, no matter what OS environment I was in.

I also wanted some more choices for getting things done than the default thread approach. This is why I modularized the milter itself to working as a traditional threading milter wherein each client connection gets its own thread, but I also added factories for single thread asynchronous operation and forking. Do note that the threaded version of python-libmilter is constrained by the GIL, whereas py-milter is not since the threading is implemented in C. If you really want to maximize speed, py-milter might be for you, but be aware that I've been able to get excellent performance from this library in production environments with very heavy processing load (evaluating many regular expressions) using the forking factory.

Note that this documentation assumes a good working knowledge of the SMTP protocol as defined in RFC 2822.

Getting It

You have a couple of options for getting this package. You can head on over to the Github page and download or clone it from there.

You can also use pip or easy_install to install it.

pip install python-libmilter

NOTE: The RPMs are not guaranteed to be the latest versions. Check Github to find out what the latest version is. That said, I will try and keep these up to date.

python-libmilter-1.0.1-1.noarch.rpm
python:python-libmilter-1.0.1-1.src.rpm

Donations

By no means should anyone feel they have to donate. However, if you've used this stuff to save some time, prevent head shaped holes in the wall, raise your children or you just think this is awesome and want to contribute to my Surly fund (that stuff isn't cheap, but it's soooooo good), you can click the Paypal donate button below and toss me a couple of dollars. I've heard from others who have donated that they were visited by flying monkeys that spoke Icelandic, and were subsequently showered with fantastic gifts from said monkeys. I don't know if this is true, or if drug use was involved, but it still sounds pretty awesome.

<html> <form action=“https://www.paypal.com/cgi-bin/webscr” method=“post”> <input type=“hidden” name=“cmd” value=“_s-xclick”> <input type=“hidden” name=“hosted_button_id” value=“83ENX2ZV8E694”> <input type=“image” src=“https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif” border=“0” name=“submit” alt=“PayPal - The safer, easier way to pay online!”> <img alt=“” border=“0” src=“https://www.paypalobjects.com/en_US/i/scr/pixel.gif” width=“1” height=“1”> </form> </html>

Bug Reports, Issues and Contributions

Please submit any and all bug reports/feature requests to the Github page.

If you wish to contribute code patches, please use the repository fork and pull request mechanisms of Github.

If you have any other questions, you can always email me directly at admin@splitstreams.com.

The Library

You will find that the library file is split up into sections defining the constants for client server options and flags. Note that there are multiple milter versions (2, 3, 4, 6) and not everything is available in earlier versions than 6. See libmilter documentation for specifics on what is available in what versions. I will show where the version 2 compatibility ends so that if you wish to write your app to be compatible with all milter versions, you can do so.

Constants

SMFIF_*

These are flags that are set defining mutations that we wish to make in our app. This first set is compatible with version 2 of the milter protocol.

SMFIF_ADDHDRS States that we may add headers to the email
SMFIF_CHGBODY States that we may replace the body
SMFIF_ADDRCPT States that we may add recipients
SMFIF_DELRCPT States that we may delete recipients
SMFIF_CHGHDRS States that we may change/delete headers (note that this is different from the body)
SMFIF_QUARANTINE States that we may quarantine the message (how this is done is up to the MTA. Postfix will put the message in its HOLD queue)

This next set is available in later milter protocol versions.

SMFIF_CHGFROM States that we may replace the envelope sender
SMFIF_ADDRCPT_PAR States that we may add recipients + args
SMFIF_SETSYMLIST We may send macro names (CURRENTLY UNSUPPORTED)

There are also some convenience flags.

SMFIF_ALLOPTS_V2 This sets all flags from the first section
SMFIF_ALLOPTS_V6 This sets all flags from the second section
SMFIF_ALLOPTS This sets all flags

SMFIP_*

These are protocol options, and can be set in a couple of different ways. The first way is to simply set them when setting up your factory (TODO: link to the factory section). The other way, at least for the SMFIP_NO* and SMFIP_NR_* options, is to use decorators. That will be explained later. First, we will define the protocol options.

Again, this first section is available to version 2 of the milter protocol.

SMFIP_NOCONNECT Tell the MTA we don't want connection info
SMFIP_NOHELO Tell the MTA we don't want HELO info
SMFIP_NOMAIL Tell the MTA we don't want MAIL info
SMFIP_NORCPT Tell the MTA we don't want RCPT info
SMFIP_NOBODY Tell the MTA we don't want the body
SMFIP_NOHDRS Tell the MTA we don't want the headers
SMFIP_NOEOH Tell the MTA we don't want the end of headers notification

These other protocol options are defined for new versions of the milter protocol than version 2.

SMFIP_NR_HDR Tell the MTA we don't reply to the headers
SMFIP_NOUNKNOWN Tell the MTA we don't want any unknown commands
SMFIP_NODATA Tell the MTA we don't want the DATA command sent to us
SMFIP_RCPT_REJ Tell the MTA we want the rejected recipients
SMFIP_NR_CONN Tell the MTA we don't reply to the headers
SMFIP_NR_HELO Tell the MTA we don't reply to HELO info
SMFIP_NR_MAIL Tell the MTA we don't reply to MAIL info
SMFIP_NR_RCPT Tell the MTA we don't reply to RCPT info
SMFIP_NR_DATA Tell the MTA we don't reply to the DATA command
SMFIP_NR_UNKN Tell the MTA we don't reply to UNKNOWNs
SMFIP_NR_EOH Tell the MTA we don't reply to the end of headers
SMFIP_NR_BODY Tell the MTA we don't reply to the body chunks
SMFIP_HDR_LEADSPC Header value has leading space

And just like the OPTS, there are some convenience variables for setting all protocol options.

SMFIP_ALLPROTOS_V2 Set all protocol options for version 2 of the protocol
SMFIP_ALLPROTOS_V6 Set all the protocol options for everything post version 2
SMFIP_ALLPROTOS Sets all protocol options

Response Constants

These are what you use to reply to the MTA with what you want it to do currently. The default is to just reply with a CONTINUE (do nothing). You should use these as the return value from your callbacks.

Example:

import libmilter
...
...
def eob(self , cmdDict):
    return libmilter.CONTINUE
...
...
Constant Description
ACCEPT Accept this message
CONTINUE Do nothing. Just continue processing the message
REJECT Reject this message/recipient/from address, etc.
TEMPFAIL Temporarily fail this message/recipient/from address, etc.
DISCARD Discard the message
CONN_FAIL Cause a connection failure
SHUTDOWN 421: shutdown (internal to MTA)

All the Rest

All the rest of the standard constants are defined as well. I've only documented the ones above as these are the only ones directly used.

Overridable Callbacks

Here is the list of the overridable callbacks. Any of these can be optionally overridden in a subclass of the MilterProtocol base class.

Note that some of these callbacks include a cmdDict argument. The cmdDict is a dictionary of everything in a fairly raw format that was sent by the MTA for this particular routine. What is contained in it varies depending on your MTA and how it is configured.

As an aside, all this information is also available with a pydoc libmilter.

connect(self , hostname , family , ip , port , cmdDict)

This gets the connection info:

Type Name Description
str hostname The reverse hostname of the connecting ip
str family The IP family (L=unix , 4=ipv4 , 6=ipv6 , U=unknown)
str ip The IP of the connecting client
int port The port number of the connecting client
dict cmdDict The raw dictionary of items sent by the MTA

helo(self , heloname)

This gets the HELO string sent by the client

Type Name Description
str heloname What the client HELOed as

mailFrom(self , frAddr , cmdDict)

This gets the MAIL FROM envelope address

Type Name Description
str frAddr The envelope from address
dict cmdDict The raw dictionary of items sent by the MTA

rcpt(self , recip , cmdDict)

This gets the RCPT TO envelope address

Type Name Description
str recip The envelope recipient address
dict cmdDict The raw dictionary of items sent by the MTA

header(self , key , val , cmdDict)

This gets one header from the email at a time. The “key” is the LHS of the header and the “val” is RHS.

ex.: key=“Subject” , val=“The subject of my email”

Type Name Description
str key The header name
str val The header value
dict cmdDict The raw dictionary of items sent by the MTA

eoh(self , cmdDict)

This tells you when all the headers have been received

Type Name Description
dict cmdDict The raw dictionary of items sent by the MTA

data(self , cmdDict)

This is called when the client sends DATA

Type Name Description
dict cmdDict The raw dictionary of items sent by the MTA

body(self , chunk , cmdDict)

This gets a chunk of the body of the email from the MTA. This will be called many times for a large email.

Type Name Description
str chunk A chunk of the email's body
dict cmdDict The raw dictionary of items sent by the MTA

eob(self , cmdDict)

This signals that the MTA has sent the entire body of the email. This is the callback where you can use modification methods, such as addHeader(), delRcpt(), etc. If you return CONTINUE from this method, it will be the same as an returning ACCEPT.

Type Name Description
dict cmdDict The raw dictionary of items sent by the MTA

close(self)

Here, you can close any open resources.

NOTE: this method is always called when everything is complete.

abort(self)

This is called when an ABORT is received from the MTA.

NOTE: Postfix will send an ABORT at the end of every message.

Modification Methods

These are all the methods that can only be used in the eob() callback in your milter to modify the message/senders/recipients themselves. Note that your MTA must support these methods for them to be used. You also must tell the MTA you are going to use these in your application with the appropriate SMFIF_* options passed into the Factory.

addRcpt(self , rcpt , esmtpAdd='')

This will tell the MTA to add a recipient to the email.

There are essentially 2 different ways you can use this. The most basic way is to just call it with a recipient address:

self.addRcpt('someone@example.com')

That will add an envelope recipient to the message. This can be very useful if you silently want to send certain messages to their another email account, perhaps for spam analysis. To use this, you must set the SMFIF_ADDRCPT option.

The second way is to call this and set the esmtpAdd variable as well. The esmtpAdd variable would be set to optional ESMTP parameters to the add recipient method.
IMPORTANT: If you want to use esmtpAdd to add additional ESMTP parameters, you have to set the SMFIF_ADDRCPT_PAR option instead of (or in addition to) the SMFIF_ADDRCPT option.

delRcpt(self , rcpt)

This will tell the MTA to delete a recipient from the email

NOTE: The recipient address must be EXACTLY the same as one of the addresses received in the rcpt() callback.

You must have the SMFIF_DELRCPT option set to use this.

replBody(self , body)

This will replace the body of the email with a new body. This is how you would make changes to the body of an email. It is up to you to store the email, modify it and send the entire body back to the MTA with this method.

You must have the SMFIF_CHGBODY option set to use this.

addHeader(self , key , val)

This will add a header to the email in the form: “key: val”

You must have the SMFIF_ADDHDRS option set to use this.

chgHeader(self , key , val='' , index=1)

This will change a header in the email. The key should be exectly what was received in header(). If val is empty, the header will be removed. index refers to which header to remove in the case that there are multiple headers with the same key (Received: is one example). Note that index starts at “1”.

You must have the SMFIF_CHGHDRS option set to use this.

quarantine(self , msg='')

This tells the MTA to quarantine the message (put it in the HOLD queue in Postfix). You can optionally set a msg to be potentially logged by the MTA.

You must have the SMFIF_QUARANTINE option set to use this.

setReply(self , rcode , xcode , msg)

Sets the reply that the MTA will use for this message.
The rcode is the 3 digit code to use (ex. 554 or 250).
The xcode is the xcode part of the reply (ex. 5.7.1 or 2.1.0). The msg is the text response.

Example:

self.setReply(555 , '5.7.0' , 'Monkey poo has just been flung at you')

chgFrom(self , frAddr , esmtpAdd='')

This tells the MTA to change the envelope From address, with optional ESMTP extensions in esmtpAdd.

You must have the SMFIF_CHGFROM option set to use this.

skip(self)

This tells the MTA that we don't want any more of this type of callback.

Unlike the other modifiers, THIS CAN ONLY BE CALLED FROM THE body() callback!!.

You must have the SMFIP_SKIP option set to use this.

Factories

When you are building your own milter, you will have to choose a factory to use to run your milter. There are 3 different factories built in, they are:

AsyncFactory A single threaded, single process asynchronous factory (not recommended)
ThreadFactory Spawns a thread per connection from the MTA. Works well with minimal processing in your milter.
ForkFactory Spawns a process for each connection from the MTA. Works the best when you don't need to share global resources

When creating a factory object, you do so by calling the factory class (AsyncFactory is used as the example here, but all the factories are the same) with the following options.

async = AsyncFactory(socketStr , MyMilter , smfifOptions , tcpListenQueue , sockChmod)

The arguments are as follows:

Arg Type Description
socketStr str This is a string representing the listen socket. It is either of the form inet:<ip>:<port> or the fullpath to where a unix domain socket should be created
MyMilter libmilter.MilterProtocol This should be your subclass implementation of libmilter.MilterProtocol
smfifOptions int This should be any SMFIF_* options you wish to set on your milter. You should bitwise OR the SMFIF_* options together. This is optional and the default is zero (no options set)
tcpListenQueue int The TCP listen queue for the socket. Optional, with a default of 50
sockChmod int What to chmod the unix domain socket to after creation. Optional, with a default of 0666. Obviously, this is ignored when you are creating an “inet” socket.

AsyncFactory

This factory is somewhat modeled after a Twisted implementation, though with some differences. This will run everything, by default, in a single thread and single process. This means that things must return very quickly (there is a way around this). This is not the recommended factory to use if you are doing anything beyond trivial checks of the message. The advantage to this factory, however, is that it makes sharing of global resources very easy.

The “single thread” model can be circumvented slightly here with the use of a decorator called callInThread that you can use with certain callbacks to send some more intensive processing to another thread, while keeping the main milter thread responsive. Here is a quick example of how to use this:

import libmilter
class MyMilter(libmilter.MilterProtocol):
...
...
    @libmilter.callInThread
    def connect(self , hostname , family , ip , port , cmdDict):
        # Run checks on the IP.  Perhaps this includes RBLs, a local database check, etc.
        if processIP(ip):
            return libmilter.CONTINUE
        else:
            return libmilter.REJECT

NOTE: No “mixin” class needs to be inherited with this factory's use.

ThreadFactory

This is a factory, wherein each connection from the MTA ends up in it's own thread. This is exactly how the C libmilter works.

This works well, with one caveat, the Python GIL. If you are making a single external call which can take a long time and is not considered blocking (like IO), that call can make your entire daemon unresponsive. An example of this would be a complex regular expression being run that can potentially take a long time to return (I found this out the hard way). If you aren't doing this kind of processing, and especially if you need to share global resources between milter instances, this is probably your best choice. If you need to run complex regular expressions and don't want to block incoming connections from your MTA, use the ForkFactory instead.

NOTE: If you choose use this, your milter protocol class implementation must inherit the ThreadMixin class as well.

Example:

import libmilter
class MyMilter(libmilter.ThreadMixin , libmilter.MilterProtocol):
...

ForkFactory

This is a factory that will fork a child process for each incoming connection from the MTA. This will add some more overhead on your machine, but is recommended for the best concurrency. See the description of ThreadFactory as to why this is.

NOTE: If you choose use this, your milter protocol class implementation must inherit the ForkMixin class as well.

Example:

import libmilter
class MyMilter(libmilter.ForkMixin , libmilter.MilterProtocol):
...

Example

See the examples directory in the distribution for a complete test milter example using the ForkFactory.

programming/python/python-libmilter.1330714768.txt.gz · Last modified: 2012/03/02 18:59 by jay