StuffIveLearned.org

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Python Prefork Server ======


===== What is it? =====
The ''py-prefork-server'' project is a simple TCP/UDP prefork server library for Python.  It is loosely modelled on the excellent Perl library [[http://search.cpan.org/~rhandom/Net-Server-0.99/lib/Net/Server/PreFork.pm|Net::Server::PreFork]].  Where the Perl library is a bit of an "everything and the kitchen sink" approach, ''py-prefork-server'' only manages the network connections and child process distribution.  Unlike [[http://search.cpan.org/~rhandom/Net-Server-0.99/lib/Net/Server/PreFork.pm|Net::Server::PreFork]], you will have to manage the daemonization, logging, pidfile, privilege dropping, etc. yourself.  This was a design decision that makes it a bit more work on the programmer's part, but keeps it very flexible in how you use it.

Speaking of design decisions, another one that I made was not to make much use of the [[http://docs.python.org/library/multiprocessing.html|multiprocessing]] module that is now part of the Python standard library.  This was done on purpose because, somewhat unfortunately, I use FreeBSD primarily in my job environment and the semaphore implementation on FreeBSD is not wholly compatible with the Python [[http://docs.python.org/library/multiprocessing.html|multiprocessing]] module.  I wanted this to be a usable library across *nix environments as much as possible, which means that I was not, unfortunately, able to use semaphores.

All that aside, this seems to perform quite well in initial testing.

===== License =====
This project is licensed under the GPLv3.  A copy of the license is included in the source and is available at [[http://www.gnu.org/licenses/gpl-3.0.txt]].

===== Downloading =====
You can download this, or even clone the repository, over at [[https://github.com/crustymonkey/py-prefork-server|Github]].

You can also install this directly using ''pip'' or ''easy_install''.
<code bash>
pip install py-prefork-server
</code>
**NOTE:** The RPMs are **not** guaranteed to be the latest versions.  Check [[https://github.com/crustymonkey/py-prefork-server|Github]] to find out what the latest version is.  That said, I will try and keep these up to date.

{{:programming:python:py-prefork-server-0.1.2-1.noarch.rpm|py-prefork-server-0.4.0-1.noarch.rpm}} \\
{{:programming:python:py-prefork-server-0.1.2-1.src.rpm|py-prefork-server-0.4.0-1.src.rpm}}

===== Issues and Feature Requests =====
Please use the bug tracker built in to [[https://github.com/crustymonkey/py-prefork-server|Github]] for all bug reports and feature requests

===== Code Contributions =====
I'm always open to code contributions.  Please use the "fork" and "pull request" mechanisms on [[https://github.com/crustymonkey/py-prefork-server|Github]] for this.
===== Installing =====
To manually install the package:
<code>
# tar -xvzf py-prefork-server-X.X.X.tar.gz
# cd py-prefork-server-*
# python setup.py install
</code>

Or install from PyPI:
<code>
pip install py-prefork-server
</code>
===== Usage =====
For the most up to date information:

<code bash>
pydoc preforkserver
pydoc preforkserver.BaseChild
pydoc preforkserver.Manager
</code>

For a complete example, see the ''examples'' directory in the release.  With that said, lets get on to what you will be primarily using, the ''BaseChild'' class.

==== BaseChild ====
This is the class you will inherit from when you create your implementation.  Your implementation will actually be handling the client connections.  There are some hooks here you can use at different stages of the connection, as well as an allow/deny mechanism.  Your basic class is going to look like this
<code python>
import preforkserver

class MyChild(preforkserver.BaseChild):
    def process_request(self):
        # handle the connection here
</code>
One important thing to remember is that any instance variables that are set here will have to be done so with the idea that each instance will handle a number of connections (by default, as you can set by the max_requests argument when creating a Manager instance).  This will be discussed later.

=== Instance Variables ===
There are some class instance variables that are set for your use.  You can access these in any of the hooks.  You will use the ''self.conn'' variable directly to communicate with your client in a ''TCP'' server.

^ Name ^ Type ^ Description ^
| self.protocol | str | This will be either "tcp" or "udp" |
| self.requests_handled | int | This is the number of requests this child has handled |
| self.conn | socket object or str | The socket object if this is a tcp server, otherwise this will be the actual payload of the udp packet |
| self.address | tuple(str , int) | An address tuple containing (ip , port) |
| self.closed | boolean | A boolean, mainly for internal use, which says whether this child has been set to be closed |
| self.error | str | A string error message, if set |

=== Hooks ===
== initialize(self) ==
Rather than reimplementing __init__, which you can do instead, you can just override this and setup variables and such that you need to set up.  This is the recommended approach.  Note that this is only called once, when the class is initialized.  This is **NOT** called on each new connection.  Use ''post_accept()'' for that purpose.

== pre_bind(self) ==
This hook is called before the main socket is created and bound to the ip:port.  This is similar to the ''initialize()'' hook in the child class.  You can use this to set up global variables, etc.  Note that this will **only** be called if you are using reuse_port as the socket will be bound in the child process.

== post_bind(self) ==
As you might have guessed, this is called right after the ''accept()'' socket has been created and bound.  Note that this will **only** be called if you are using reuse_port as the socket will be bound in the child process.

== post_accept(self) ==
''self.conn'' and ''self.address'' are set before this is called, as a new connection has been established.  You can make any modifications/setup before ''process_request()'' is called.

== allow_deny(self) ==
You can use this hook to refuse the connection based on the ''self.conn'' and ''self.address'' variables that have been set.  Return ''True'' (default) here to accept the connection and ''False'' to deny it and close the connection.  This is especially useful for IP based filtering.  See ''request_denied()'' for sending messages back to the client before the socket is closed.  Note that this is the only hook where your return value matters.

== request_denied(self) ==
If you deny the connection in allow_deny(), you can send a message using this callback before the connection is closed.

== process_request(self) ==
This is where you are processing the actual request.  You should use your self.conn socket to send and receive data from your client in this hook.

Remember, if this is a udp server, self.conn will be a string with the actual packet payload.  If you have a udp server, and you wish to respond, you can use the resp_to() method built into BaseChild.  It accepts the message responds to the current udp client.

<code python>
def process_request(self):
    data = self.conn
    self.resp_to('Received data: %s\n' % data)
</code>

== post_process_request(self) ==
This is called after the connection is closed.  You can perform any maintenance/cleanup or post connection processing here.

== shutdown(self) ==
This is called when the child is exiting.  This can be because it has served its maximum number of requests, the ''Manager'' was told to close, or an error has occured.  If an error has occured, ''self.error'' will be set.

Use this to do an pre-close cleanup, like possibly closing open files or a database connection.

==== Manager ====
The ''Manager'' class is the controller that handles all the child processes.  This is a fully implemented class that, in just about all cases, you will just create an instance of and call its ''run()'' method.  If you have some special setup needs there are a number of hooks that can be overridden in a subclass for setup.  Some of the more useful methods to override are those for handling signals.  We will get to that later.

First, let's take a look at the ''__init__()'' signature.  The majority of these arguments are optional, and in fact, the only one that is required is your implementation of ''BaseChild''.

=== The __init__ Signature ===
<code python>
def __init__(self , child_class , max_servers=20 , min_servers=5 ,
        min_spare_servers=2 , max_spare_servers=10 , max_requests=0 ,
        bind_ip='127.0.0.1' , port=10000 , protocol='tcp' , listen=5 ,
        reuse_port=False):
</code>
Here is short description of each of those variables.
^ Name ^ Type ^ Description ^
| child_class | BaseChild | An implentation of BaseChild to define the child processes |
| max_cervers | int | Maximum number of children that can exist at any time |
| min_servers | int | Minimum number of children to have |
| min_spare_servers | int | Minimum number of spare children to have |
| max_spare_servers | int | Maximum number of spare children to have |
| max_requests | int | Maximum number of requests each child should handle.  Zero is unlimited and the default. |
| bind_ip | str | The IP address to bind to |
| port | int | The port number to bind to |
| protocol | str | The protocol to use (tcp or udp) |
| listen | int | TCP listen queue backlog |
| reuse_port | bool | If set, and available, SO_REUSEPORT will be used to bind the socket in the child |
For anyone who has set up an Apache prefork server (or used Perl's ''Net::Server::PreFork''), this should look pretty familiar.

=== Hooks ===
Of the following hooks, most, with the exception of the signal hooks, are not going to be very useful to you.  The signal hooks can be overridden to perform special actions on a ''HUP'', ''INT'' or ''TERM''.  

Here they are with a brief description.

== pre_bind(self) ==
This hook is called before the main socket is created and bound to the ip:port.  This is similar to the ''initialize()'' hook in the child class.  You can use this to set up global variables, etc.  Note that this will **not** be called if you are using reuse_port as the socket will be bound in the child process.

== post_bind(self) ==
As you might have guessed, this is called right after the ''accept()'' socket has been created and bound.  Note that this will **not** be called if you are using reuse_port as the socket will be bound in the child process.

== pre_signal_setup(self) ==
This is called before the signal handlers are set up

== post_signal_setup(self) ==
This is called after the signal handlers have been set.  You can override the default signal handlers if you like.  More on that below.

== pre_init_children(self) ==
This is called before the child processes are initialized.

== post_init_children(self) ==
This is called after the child processes are initialized.

== pre_loop(self) ==
This is the last hook before the main server loop takes over. Any last minute setup items you wish to do should be done here.

== pre_server_close(self) ==
This is called before the server shuts down.  Any cleanup you wish to take care of before termination should be done here.

== hup_handler(self , frame , num) ==
This handles a SIGHUP.  If you have a config for your server, you could reload that here.  By default, this just ignores the signal.

== int_handler(self , frame , num) ==
This handles a SIGINT.  The default is set an internal "stop" event to gracefully shutdown.  If you override this, call the ''super()'' so the graceful shutdown happens.

== term_handler(self , frame , num) ==
By default, this does exactly the same thing as ''intHandler'' with a SIGTERM.

===== Example =====
There is a full, working example in the distributions ''examples'' directory.