StuffIveLearned.org

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== python-libmilter ======

===== About =====
python-libmilter started as a project to maximize the portability of libmilter.  I know about, and have used, the excellent CPython implementation by Stuart D. Gathman [[http://bmsi.com/python/milter.html|py-milter]].  Though I liked this library for it's thin wrapper around the C libmilter library (and thereby speed), I had some issues with getting it to install, having to find and download the appropriate source code for libmilter version installed, etc.  This is why I went with a ground up implementation of libmilter in pure Python.  I want something I could just "drop in" to a project and have a milter up and running in almost no time, no matter what OS environment I was in.  

I also wanted some more choices for getting things done than the default thread approach.  This is why I modularized the milter itself to working as a traditional threading milter wherein each client connection gets its own thread, but I also added factories for single thread asynchronous operation and forking.  Do note that the threaded version of python-libmilter is constrained by the GIL, whereas [[http://bmsi.com/python/milter.html|py-milter]] is not since the threading is implemented in C.  If you really want to maximize speed, [[http://bmsi.com/python/milter.html|py-milter]] might be for you, but be aware that I've been able to get excellent performance from this library in production environments with very heavy processing load (evaluating many regular expressions) using the forking factory.

Note that this documentation assumes a good working knowledge of the SMTP protocol as defined in RFC 2822.

===== Getting It =====
You have a couple of options for getting this package.  You can head on over to the [[https://github.com/crustymonkey/python-libmilter|Github page]] and download or clone it from there.  

You can also use ''pip'' or ''easy_install'' to install it.
<code bash>
pip install python-libmilter
</code>

**NOTE:** The RPMs are **not** guaranteed to be the latest versions.  Check [[https://github.com/crustymonkey/python-libmilter|Github]] to find out what the latest version is.  That said, I will try and keep these up to date.

{{:programming:python:python-libmilter-1.0.1-1.noarch.rpm|python-libmilter-1.0.1-1.noarch.rpm}} \\
{{:programming:python:python-libmilter-1.0.1-1.src.rpm|python:python-libmilter-1.0.1-1.src.rpm}}

===== Bug Reports, Issues and Contributions =====
Please submit any and all bug reports/feature requests to the [[https://github.com/crustymonkey/python-libmilter|Github page]].

If you wish to contribute code patches, please use the repository fork and pull request mechanisms of [[http://github.com|Github]].

If you have any other questions, you can always email me directly at admin@splitstreams.com.

===== The Library =====
You will find that the library file is split up into sections defining the constants for client server options and flags.  Note that there are multiple milter versions (2, 3, 4, 6) and not everything is available in earlier versions than 6.  See libmilter documentation for specifics on what is available in what versions.  I will show where the version 2 compatibility ends so that if you wish to write your app to be compatible with all milter versions, you can do so.

==== Constants ====

=== SMFIF_* ===
These are flags that are set defining mutations that we wish to make in our app.  This first set is compatible with version 2 of the milter protocol.

^ SMFIF_ADDHDRS | States that we may add headers to the email |
^ SMFIF_CHGBODY | States that we may replace the body |
^ SMFIF_ADDRCPT | States that we may add recipients |
^ SMFIF_DELRCPT | States that we may delete recipients |
^ SMFIF_CHGHDRS | States that we may change/delete headers (note that this is different from the body) |
^ SMFIF_QUARANTINE | States that we may quarantine the message (how this is done is up to the MTA.  Postfix will put the message in its HOLD queue) |

This next set is available in later milter protocol versions.

^ SMFIF_CHGFROM | States that we may replace the envelope sender |
^ SMFIF_ADDRCPT_PAR | States that we may add recipients + args |
^ SMFIF_SETSYMLIST | We may send macro names (CURRENTLY UNSUPPORTED) |

There are also some convenience flags.

^ SMFIF_ALLOPTS_V2 | This sets all flags from the first section |
^ SMFIF_ALLOPTS_V6 | This sets all flags from the second section |
^ SMFIF_ALLOPTS | This sets all flags |

=== SMFIP_* ===
These are protocol options, and can be set in a couple of different ways.  The first way is to simply set them when setting up your factory (TODO: link to the factory section).  The other way, at least for the ''SMFIP_NO*'' and ''SMFIP_NR_*'' options, is to use decorators.  That will be explained later.  First, we will define the protocol options.

Again, this first section is available to version 2 of the milter protocol.

^ SMFIP_NOCONNECT | Tell the MTA we don't want connection info |
^ SMFIP_NOHELO | Tell the MTA we don't want HELO info |
^ SMFIP_NOMAIL | Tell the MTA we don't want MAIL info |
^ SMFIP_NORCPT | Tell the MTA we don't want RCPT info |
^ SMFIP_NOBODY | Tell the MTA we don't want the body |
^ SMFIP_NOHDRS | Tell the MTA we don't want the headers |
^ SMFIP_NOEOH | Tell the MTA we don't want the end of headers notification |

These other protocol options are defined for new versions of the milter protocol than version 2.

^ SMFIP_NR_HDR | Tell the MTA we don't reply to the headers |
^ SMFIP_NOUNKNOWN | Tell the MTA we don't want any unknown commands |
^ SMFIP_NODATA | Tell the MTA we don't want the DATA command sent to us |
^ SMFIP_RCPT_REJ | Tell the MTA we want the rejected recipients |
^ SMFIP_NR_CONN | Tell the MTA we don't reply to the headers |
^ SMFIP_NR_HELO | Tell the MTA we don't reply to HELO info |
^ SMFIP_NR_MAIL | Tell the MTA we don't reply to MAIL info |
^ SMFIP_NR_RCPT | Tell the MTA we don't reply to RCPT info |
^ SMFIP_NR_DATA | Tell the MTA we don't reply to the DATA command |
^ SMFIP_NR_UNKN | Tell the MTA we don't reply to UNKNOWNs |
^ SMFIP_NR_EOH | Tell the MTA we don't reply to the end of headers |
^ SMFIP_NR_BODY | Tell the MTA we don't reply to the body chunks |
^ SMFIP_HDR_LEADSPC | Header value has leading space |

And just like the OPTS, there are some convenience variables for setting all protocol options.

^ SMFIP_ALLPROTOS_V2 | Set all protocol options for version 2 of the protocol |
^ SMFIP_ALLPROTOS_V6 | Set all the protocol options for everything post version 2 |
^ SMFIP_ALLPROTOS | Sets all protocol options |

=== Response Constants ===
These are what you use to reply to the MTA with what you want it to do currently.  The default is to just reply with a CONTINUE (do nothing).  You should use these as the ''return'' value from your callbacks.

Example:
<code python>
import libmilter
...
...
def eob(self , cmdDict):
    return libmilter.CONTINUE
...
...
</code>

^ Constant ^ Description ^
| ACCEPT | Accept this message |
| CONTINUE | Do nothing.  Just continue processing the message |
| REJECT | Reject this message/recipient/from address, etc. |
| TEMPFAIL | Temporarily fail this message/recipient/from address, etc. |
| DISCARD | Discard the message |
| CONN_FAIL | Cause a connection failure |
| SHUTDOWN | 421: shutdown (internal to MTA) |

=== All the Rest ===
All the rest of the standard constants are defined as well.  I've only documented the ones above as these are the only ones directly used.

==== Overridable Callbacks ====
Here is the list of the overridable callbacks.  Any of these can be optionally overridden in a subclass of the ''MilterProtocol'' base class.

Note that some of these callbacks include a ''cmdDict'' argument.  The ''cmdDict'' is a dictionary of everything in a fairly raw format that was sent by the MTA for this particular routine.  What is contained in it varies depending on your MTA and how it is configured.

As an aside, all this information is also available with a ''pydoc libmilter''.

=== connect(self , hostname , family , ip , port , cmdDict) ===
This gets the connection info:

^ Type ^ Name ^ Description ^
| str | hostname  | The reverse hostname of the connecting ip |
| str | family | The IP family (L=unix , 4=ipv4 , 6=ipv6 , U=unknown) |
| str | ip| The IP of the connecting client |
| int | port | The port number of the connecting client |
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== helo(self , heloname) ===
This gets the HELO string sent by the client

^ Type ^ Name ^ Description ^
| str | heloname | What the client HELOed as |

=== mailFrom(self , frAddr , cmdDict) ===
This gets the MAIL FROM envelope address

^ Type ^ Name ^ Description ^
| str | frAddr | The envelope from address |
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== rcpt(self , recip , cmdDict) ===
This gets the RCPT TO envelope address

^ Type ^ Name ^ Description ^
| str | recip | The envelope recipient address |
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== header(self , key , val , cmdDict) ===
This gets one header from the email at a time.  The "key" is the LHS of the header and the "val" is RHS.

ex.: key="Subject" , val="The subject of my email"

^ Type ^ Name ^ Description ^
| str | key | The header name |
| str | val | The header value |
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== eoh(self , cmdDict) ===
This tells you when all the headers have been received

^ Type ^ Name ^ Description ^
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== data(self , cmdDict) ===
This is called when the client sends DATA

^ Type ^ Name ^ Description ^
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== body(self , chunk , cmdDict) ===
This gets a chunk of the body of the email from the MTA. This will be called many times for a large email.

^ Type ^ Name ^ Description ^
| str | chunk | A chunk of the email's body |
| dict | cmdDict | The raw dictionary of items sent by the MTA |

===  eob(self , cmdDict) ===
This signals that the MTA has sent the entire body of the email. This is the callback where you can use [[#modification methods]], such as addHeader(), delRcpt(), etc.  If you return CONTINUE from this method, it will be the same as an returning ACCEPT.

^ Type ^ Name ^ Description ^
| dict | cmdDict | The raw dictionary of items sent by the MTA |

=== close(self) ===
Here, you can close any open resources. 

**NOTE:** this method is **always** called when everything is complete.

=== abort(self) ===
This is called when an ABORT is received from the MTA.

**NOTE:** Postfix will send an ABORT at the end of every message.

==== Modification Methods ====
These are all the methods that can **only be used in the eob()** callback in your milter to modify the message/senders/recipients themselves.  Note that your MTA must support these methods for them to be used.  You also must tell the MTA you are going to use these in your application with the appropriate [[#SMFIF_*]] options passed into the [[#Factories|Factory]].

=== addRcpt(self , rcpt , esmtpAdd='') ===
This will tell the MTA to add a recipient to the email.

There are essentially 2 different ways you can use this.  The most basic way is to just call it with a recipient address:
<code python>
self.addRcpt('someone@example.com')
</code>
That will add an envelope recipient to the message.  This can be very useful if you silently want to send certain messages to their another email account, perhaps for spam analysis.  To use this, you must set the ''SMFIF_ADDRCPT'' option.

The second way is to call this and set the ''esmtpAdd'' variable as well.  The ''esmtpAdd'' variable would be set to optional ESMTP parameters to the add recipient method. \\
**IMPORTANT:**  If you want to use ''esmtpAdd'' to add additional ESMTP parameters, you have to set the ''SMFIF_ADDRCPT_PAR'' option instead of (or in addition to) the ''SMFIF_ADDRCPT'' option.

=== delRcpt(self , rcpt) ===
This will tell the MTA to delete a recipient from the email

**NOTE:** The recipient address must be EXACTLY the same as one of the addresses received in the rcpt() callback.

You must have the ''SMFIF_DELRCPT'' option set to use this.

=== replBody(self , body) ===
This will replace the body of the email with a new body.  This is how you would make changes to the body of an email.  It is up to you to store the email, modify it and send the entire body back to the MTA with this method.

You must have the ''SMFIF_CHGBODY'' option set to use this.

=== addHeader(self , key , val) ===
This will add a header to the email in the form: "key: val"

You must have the ''SMFIF_ADDHDRS'' option set to use this.

=== chgHeader(self , key , val='' , index=1) ===
This will change a header in the email.  The ''key'' should be exectly what was received in header().  If ''val'' is empty, the header will be removed.  ''index'' refers to which header to remove in the case that there are multiple headers with the same ''key'' (//Received:// is one example).  Note that ''index'' starts at "1".

You must have the ''SMFIF_CHGHDRS'' option set to use this.

=== quarantine(self , msg='') ===
This tells the MTA to quarantine the message (put it in the HOLD queue in Postfix).  You can optionally set a ''msg'' to be potentially logged by the MTA.

You must have the ''SMFIF_QUARANTINE'' option set to use this.

=== setReply(self , rcode , xcode , msg) ===
Sets the reply that the MTA will use for this message.\\
The ''rcode'' is the 3 digit code to use (ex. 554 or 250).\\
The ''xcode'' is the xcode part of the reply (ex. 5.7.1 or 2.1.0).
The ''msg'' is the text response.

Example:
<code python>
self.setReply(555 , '5.7.0' , 'Monkey poo has just been flung at you')
</code>

=== chgFrom(self , frAddr , esmtpAdd='') ===
This tells the MTA to change the envelope From address, with optional ESMTP extensions in ''esmtpAdd''.

You must have the ''SMFIF_CHGFROM'' option set to use this.

=== skip(self) ===
This tells the MTA that we don't want any more of this type of callback.

Unlike the other modifiers, **THIS CAN ONLY BE CALLED FROM THE body() callback!!**.

You must have the ''SMFIP_SKIP'' option set to use this.

==== Factories ====
When you are building your own milter, you will have to choose a factory to use to run your milter.  There are 3 different factories built in, they are:

^ AsyncFactory | A single threaded, single process asynchronous factory (not recommended) |
^ ThreadFactory | Spawns a thread per connection from the MTA.  Works well with minimal processing in your milter. |
^ ForkFactory | Spawns a process for each connection from the MTA.  Works the best when you don't need to share global resources |

When creating a factory object, you do so by calling the factory class (''AsyncFactory'' is used as the example here, but all the factories are the same) with the following options.

<code python>
async = AsyncFactory(socketStr , MyMilter , smfifOptions , tcpListenQueue , sockChmod)
</code>

The arguments are as follows:
^ Arg ^ Type ^ Description ^
| socketStr | str | This is a string representing the listen socket.  It is either of the form ''inet:<ip>:<port>'' or the fullpath to where a unix domain socket should be created |
| MyMilter | libmilter.MilterProtocol | This should be your subclass implementation of ''libmilter.MilterProtocol'' |
| smfifOptions | int | This should be any [[#SMFIF_*]] options you wish to set on your milter.  You should bitwise OR the [[#SMFIF_*]] options together. This is optional and the default is zero (no options set) |
| tcpListenQueue | int | The TCP listen queue for the socket.  Optional, with a default of 50 |
| sockChmod | int | What to ''chmod'' the unix domain socket to after creation.  Optional, with a default of ''0666''.  Obviously, this is ignored when you are creating an "inet" socket. |

=== AsyncFactory ===
This factory is somewhat modeled after a Twisted implementation, though with some differences.  This will run everything, by default, in a single thread and single process.  This means that things must return very quickly (there is a way around this).  This is not the recommended factory to use if you are doing anything beyond trivial checks of the message.  The advantage to this factory, however, is that it makes sharing of global resources very easy.

The "single thread" model can be circumvented slightly here with the use of a decorator called ''callInThread'' that you can use with certain callbacks to send some more intensive processing to another thread, while keeping the main milter thread responsive.  Here is a quick example of how to use this:

<code python>
import libmilter
class MyMilter(libmilter.MilterProtocol):
...
...
    @libmilter.callInThread
    def connect(self , hostname , family , ip , port , cmdDict):
        # Run checks on the IP.  Perhaps this includes RBLs, a local database check, etc.
        if processIP(ip):
            return libmilter.CONTINUE
        else:
            return libmilter.REJECT
</code>

**NOTE:** No "mixin" class needs to be inherited with this factory's use.

=== ThreadFactory ===
This is a factory, wherein each connection from the MTA ends up in it's own thread.  This is exactly how the C libmilter works.

This works well, with one caveat, the Python GIL.  If you are making a single external call which can take a long time and is not considered blocking (like IO), that call can make your entire daemon unresponsive.  An example of this would be a complex regular expression being run that can potentially take a long time to return (I found this out the hard way).  If you aren't doing this kind of processing, and especially if you need to share global resources between milter instances, this is probably your best choice.  If you need to run complex regular expressions and don't want to block incoming connections from your MTA, use the [[#ForkFactory]] instead.

**NOTE:** If you choose use this, your milter protocol class implementation **must** inherit the ''ThreadMixin'' class as well.

Example:
<code python>
import libmilter
class MyMilter(libmilter.ThreadMixin , libmilter.MilterProtocol):
...
</code>

=== ForkFactory ===
This is a factory that will fork a child process for each incoming connection from the MTA.  This will add some more overhead on your machine, but is recommended for the best concurrency.  See the description of [[#ThreadFactory]] as to why this is.

**NOTE:** If you choose use this, your milter protocol class implementation **must** inherit the ''ForkMixin'' class as well.

Example:
<code python>
import libmilter
class MyMilter(libmilter.ForkMixin , libmilter.MilterProtocol):
...
</code>

===== Example =====
See the examples directory in the distribution for a complete test milter example using the [[#ForkFactory]].