| Both sides previous revisionPrevious revisionNext revision | Previous revision |
| programming:python:clcheck [2009/06/15 19:17] – crustymonkey | programming:python:clcheck [2011/03/29 20:15] (current) – [What is it?] jay |
|---|
| ====== Craigslist Search and Notify Script ====== | ====== Craigslist Search and Notify Script ====== |
| | ===== NO LONGER MAINTAINED ===== |
| | Sorry to anyone who has arrived here, but I'm no longer maintaining this script. I remembered, when initially writing it, what a pain in the ass it was to try and maintain a screen scraping app. Scraping Craigslist, I've found, is especially masochistic since the site seems to sometimes vary slightly between cities on top of the fact that once they change the underlying HTML (which is pretty awful to start with), it breaks everything. |
| | |
| | It's not the most complicated of scripts, so anyone with some Python chops should be able to still take the base and do some tweaking (mainly of the regular expressions) and get this to work. I just don't have the large amount of time to sink into supporting something that is broken by the whims of others. If, in the future, Craigslist publishes a real API of some sort, I will be happy as a clam to rewrite this app using that. Bottom line here: screen scraping sucks for anyone doing it. |
| | |
| ===== What is it? ===== | ===== What is it? ===== |
| A friend of mine asked me recently if I knew of a script that would perform a search on [[http://craigslist.org|craigslist.org]] and send him an email when there were new results to his search. He's always looking for oddball stuff and by the time //he// finds it on craigslist and emails/calls the person, the item is already gone. | A friend of mine asked me recently if I knew of a script that would perform a search on [[http://craigslist.org|craigslist.org]] and send him an email when there were new results to his search. He's always looking for oddball stuff and by the time //he// finds it on craigslist and emails/calls the person, the item is already gone. |
| |
| ===== The Script and Config File ===== | ===== The Script and Config File ===== |
| Setting this up is a piece of cake. All you have to do is get the script and the example config file, edit the config and run the script. This has been tested to work under both Windows and Linux with Python 2.6, but it should work just fine under pretty much any platform. If you run into some kind of error, run the script in debugging mode and send the output to admin@splitstreams.com so I can fix it. | Setting this up is a piece of cake. All you have to do is get the script and the example config file, edit the config and run the script. This has been tested to work under both Windows and Linux with Python 2.6, but it should work just fine under pretty much any platform. If you run into some kind of error, run the script in debugging mode and send the output to <admin@splitstreams.com> so I can fix it. |
| ==== Getting the Files ==== | ==== Getting the Files ==== |
| | === Subversion === |
| You can check out the latest script and example config from [[http://subversion.tigris.org/|Subversion]] with: | You can check out the latest script and example config from [[http://subversion.tigris.org/|Subversion]] with: |
| |
| |
| If you don't have [[http://subversion.tigris.org/|Subversion]] installed, you can just browse the to the url above, right click on each file and select "Save Link As" and save a copy of the script and config example file. On the [[http://subversion.tigris.org/|Subversion]] note, if you are on Windows, I highly suggest using [[http://tortoisesvn.tigris.org/|TortoiseSVN]] for actual SVN access. | If you don't have [[http://subversion.tigris.org/|Subversion]] installed, you can just browse the to the url above, right click on each file and select "Save Link As" and save a copy of the script and config example file. On the [[http://subversion.tigris.org/|Subversion]] note, if you are on Windows, I highly suggest using [[http://tortoisesvn.tigris.org/|TortoiseSVN]] for actual SVN access. |
| | === Download the Package === |
| | The package can be downloaded directly by {{:programming:python:clwatch-0.3.5.tar.gz|clicking here}}. |
| |
| ==== The Example Config ==== | ==== The Example Config ==== |
| ===== Usage ===== | ===== Usage ===== |
| As you can see in the above config file example, the configuration is actually pretty simple. You should set up a ''DEFAULTS'' section with just about everything there with the exception of the ''search'' option. This will limit the amount you have to repeat yourself in your actual search configs. Note that you can override anything in the search configs and it will be used instead. For example, you could use separate ''dbLoc'' directives and use a different db file for each search, if you so choose. You could also override clBase and put in ''sfbay.craigslist.org'' if you were looking to move to San Francisco and wanted to scope out houses there. | As you can see in the above config file example, the configuration is actually pretty simple. You should set up a ''DEFAULTS'' section with just about everything there with the exception of the ''search'' option. This will limit the amount you have to repeat yourself in your actual search configs. Note that you can override anything in the search configs and it will be used instead. For example, you could use separate ''dbLoc'' directives and use a different db file for each search, if you so choose. You could also override clBase and put in ''sfbay.craigslist.org'' if you were looking to move to San Francisco and wanted to scope out houses there. |
| | |
| | Listed in order of importance, 1. being the most important, here is how ''clcheck.py'' will search for config files: |
| | |
| | - ''-c'' command-line option |
| | - ''~/.clwatch.cfg'' |
| | - ''/usr/local/etc/clwatch.cfg'' |
| | - ''/etc/clwatch.cfg'' |
| | - ''./clwatch.cfg'' <- config in your current directory |
| |
| After the ''DEFAULTS'' section, the individual searches are defined by a name in brackets, []. The name can be anything you wish, but should probably be short since, if you are using the email option, it will be part of the subject. | After the ''DEFAULTS'' section, the individual searches are defined by a name in brackets, []. The name can be anything you wish, but should probably be short since, if you are using the email option, it will be part of the subject. |
| |
| ===== Any Further Questions ===== | ===== Any Further Questions ===== |
| If you have any further questions or suggestions, email me at admin@splitstreams.com | If you have any further questions or suggestions, email me at <admin@splitstreams.com>. |
| |
| | If you run into any bugs, you can either email me or open a bug report at https://bugzilla.splitstreams.com |
