The py-prefork-server project is a simple TCP/UDP prefork server library for Python. It is loosely modelled on the excellent Perl library Net::Server::PreFork. Where the Perl library is a bit of an “everything and the kitchen sink” approach, py-prefork-server only manages the network connections and child process distribution. Unlike Net::Server::PreFork, you will have to manage the daemonization, logging, pidfile, privilege dropping, etc. yourself. This was a design decision that makes it a bit more work on the programmer's part, but keeps it very flexible in how you use it.
Speaking of design decisions, another one that I made was not to make much use of the multiprocessing module that is now part of the Python standard library. This was done on purpose because, somewhat unfortunately, I use FreeBSD primarily in my job environment and the semaphore implementation on FreeBSD is not wholly compatible with the Python multiprocessing module. I wanted this to be a usable library across *nix environments as much as possible, which means that I was not, unfortunately, able to use semaphores.
All that aside, this seems to perform quite well in initial testing.
This project is licensed under the GPLv3. A copy of the license is included in the source and is available at http://www.gnu.org/licenses/gpl-3.0.txt.
You can download this, or even clone the repository, over at Github.
You can also install this directly using pip or easy_install.
pip install py-prefork-server
NOTE: The RPMs are not guaranteed to be the latest versions. Check Github to find out what the latest version is. That said, I will try and keep these up to date.
py-prefork-server-0.4.0-1.noarch.rpm
py-prefork-server-0.4.0-1.src.rpm
Please use the bug tracker built in to Github for all bug reports and feature requests
I'm always open to code contributions. Please use the “fork” and “pull request” mechanisms on Github for this.
To manually install the package:
# tar -xvzf py-prefork-server-X.X.X.tar.gz # cd py-prefork-server-* # python setup.py install
Or install from PyPI:
pip install py-prefork-server
For the most up to date information:
pydoc preforkserver pydoc preforkserver.BaseChild pydoc preforkserver.Manager
For a complete example, see the examples directory in the release. With that said, lets get on to what you will be primarily using, the BaseChild class.
This is the class you will inherit from when you create your implementation. Your implementation will actually be handling the client connections. There are some hooks here you can use at different stages of the connection, as well as an allow/deny mechanism. Your basic class is going to look like this
import preforkserver class MyChild(preforkserver.BaseChild): def process_request(self): # handle the connection here
One important thing to remember is that any instance variables that are set here will have to be done so with the idea that each instance will handle a number of connections (by default, as you can set by the max_requests argument when creating a Manager instance). This will be discussed later.
There are some class instance variables that are set for your use. You can access these in any of the hooks. You will use the self.conn variable directly to communicate with your client in a TCP server.
| Name | Type | Description |
|---|---|---|
| self.protocol | str | This will be either “tcp” or “udp” |
| self.requests_handled | int | This is the number of requests this child has handled |
| self.conn | socket object or str | The socket object if this is a tcp server, otherwise this will be the actual payload of the udp packet |
| self.address | tuple(str , int) | An address tuple containing (ip , port) |
| self.closed | boolean | A boolean, mainly for internal use, which says whether this child has been set to be closed |
| self.error | str | A string error message, if set |
Rather than reimplementing init, which you can do instead, you can just override this and setup variables and such that you need to set up. This is the recommended approach. Note that this is only called once, when the class is initialized. This is NOT called on each new connection. Use post_accept() for that purpose.
This hook is called before the main socket is created and bound to the ip:port. This is similar to the initialize() hook in the child class. You can use this to set up global variables, etc. Note that this will only be called if you are using reuse_port as the socket will be bound in the child process.
As you might have guessed, this is called right after the accept() socket has been created and bound. Note that this will only be called if you are using reuse_port as the socket will be bound in the child process.
self.conn and self.address are set before this is called, as a new connection has been established. You can make any modifications/setup before process_request() is called.
You can use this hook to refuse the connection based on the self.conn and self.address variables that have been set. Return True (default) here to accept the connection and False to deny it and close the connection. This is especially useful for IP based filtering. See request_denied() for sending messages back to the client before the socket is closed. Note that this is the only hook where your return value matters.
If you deny the connection in allow_deny(), you can send a message using this callback before the connection is closed.
This is where you are processing the actual request. You should use your self.conn socket to send and receive data from your client in this hook.
Remember, if this is a udp server, self.conn will be a string with the actual packet payload. If you have a udp server, and you wish to respond, you can use the resp_to() method built into BaseChild. It accepts the message responds to the current udp client.
def process_request(self): data = self.conn self.resp_to('Received data: %s\n' % data)
This is called after the connection is closed. You can perform any maintenance/cleanup or post connection processing here.
This is called when the child is exiting. This can be because it has served its maximum number of requests, the Manager was told to close, or an error has occured. If an error has occured, self.error will be set.
Use this to do an pre-close cleanup, like possibly closing open files or a database connection.
The Manager class is the controller that handles all the child processes. This is a fully implemented class that, in just about all cases, you will just create an instance of and call its run() method. If you have some special setup needs there are a number of hooks that can be overridden in a subclass for setup. Some of the more useful methods to override are those for handling signals. We will get to that later.
First, let's take a look at the init() signature. The majority of these arguments are optional, and in fact, the only one that is required is your implementation of BaseChild.
def __init__(self , child_class , max_servers=20 , min_servers=5 , min_spare_servers=2 , max_spare_servers=10 , max_requests=0 , bind_ip='127.0.0.1' , port=10000 , protocol='tcp' , listen=5 , reuse_port=False):
Here is short description of each of those variables.
| Name | Type | Description |
|---|---|---|
| child_class | BaseChild | An implentation of BaseChild to define the child processes |
| max_cervers | int | Maximum number of children that can exist at any time |
| min_servers | int | Minimum number of children to have |
| min_spare_servers | int | Minimum number of spare children to have |
| max_spare_servers | int | Maximum number of spare children to have |
| max_requests | int | Maximum number of requests each child should handle. Zero is unlimited and the default. |
| bind_ip | str | The IP address to bind to |
| port | int | The port number to bind to |
| protocol | str | The protocol to use (tcp or udp) |
| listen | int | TCP listen queue backlog |
| reuse_port | bool | If set, and available, SO_REUSEPORT will be used to bind the socket in the child |
For anyone who has set up an Apache prefork server (or used Perl's Net::Server::PreFork), this should look pretty familiar.
Of the following hooks, most, with the exception of the signal hooks, are not going to be very useful to you. The signal hooks can be overridden to perform special actions on a HUP, INT or TERM.
Here they are with a brief description.
This hook is called before the main socket is created and bound to the ip:port. This is similar to the initialize() hook in the child class. You can use this to set up global variables, etc. Note that this will not be called if you are using reuse_port as the socket will be bound in the child process.
As you might have guessed, this is called right after the accept() socket has been created and bound. Note that this will not be called if you are using reuse_port as the socket will be bound in the child process.
This is called before the signal handlers are set up
This is called after the signal handlers have been set. You can override the default signal handlers if you like. More on that below.
This is called before the child processes are initialized.
This is called after the child processes are initialized.
This is the last hook before the main server loop takes over. Any last minute setup items you wish to do should be done here.
This is called before the server shuts down. Any cleanup you wish to take care of before termination should be done here.
This handles a SIGHUP. If you have a config for your server, you could reload that here. By default, this just ignores the signal.
This handles a SIGINT. The default is set an internal “stop” event to gracefully shutdown. If you override this, call the super() so the graceful shutdown happens.
By default, this does exactly the same thing as intHandler with a SIGTERM.
There is a full, working example in the distributions examples directory.