Darryl Cousins

Running a HTTP Server in a Thread

2010-04-07T20:41:00.000-07:00

I'm building a shop using Django and Satchmo and I ran into an interesting problem while building tests for the Paypal payment module.

The user is sent off to Paypal to process the payment and are returned to the success page. Paypal notifies the site of payment through an 'ipn' url on the shop. Faking the call to the ipn url is easy enough in the test:

>>> postdata = dict(payment_status='Completed',
...     invoice=order.id, memo=memo, txn_id='1234',
...     mc_gross=order.total)
>>> response = client.post('/checkout/paypal/ipn/', postdata)

But the problem then arose because the ipn django view then requests verification from paypal:

def confirm_ipn_data(data, PP_URL):
    # data is the form data that was submitted to the IPN URL.

    newparams = {}
    for key in data.keys():
        newparams[key] = data[key]

    newparams['cmd'] = "_notify-validate"
    params = urlencode(newparams)

    req = urllib2.Request(PP_URL)
    req.add_header("Content-type", "application/x-www-form-urlencoded")
    fo = urllib2.urlopen(req, params)

    ret = fo.read()
    if ret == "VERIFIED":
        log.info("PayPal IPN data verification was successful.")
    else:
        log.info("PayPal IPN data verification failed.")
        log.debug("HTTP code %s, response text: '%s'" % (fo.code, ret))
        return False

    return True

The problem here is opening the url using urllib2. I couldn't use the http://testserver/ that is 'running' in the Django test code. So I tried to start up a simple HTTPServer instance but this would lock up the testrunner. Finally then I came up with the following code which does the job nicely:

# simple server to use as fake verification server
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
from threading import Thread

class PaypalHandler(BaseHTTPRequestHandler):
    """
    Simple server to return verified value for our paypal tests
    """
    def do_POST(self):
        self.send_response(200, 'OK')
        self.send_header('Content-type', 'text/html')
        self.end_headers()
        self.wfile.write( "VERIFIED" )

class RunThread(Thread):
    def __init__(self, port):
        Thread.__init__(self)
        self.server = HTTPServer(('127.0.0.1', port), PaypalHandler)
    def run(self):
        self.server.serve_forever()

server_thread = None

def start_server(port):
    global server_thread
    server_thread = RunThread(port)
    server_thread.start()

def stop_server():
    global server_thread
    server_thread.server.socket.close()

Now in my test setup I can call

def setUp(suite):
    testing.start_server(10080)

def tearDown(suite):
    testing.stop_server()

And it works as expected.

Snow Leopard and Zope3

2010-02-28T15:53:00.000-08:00

I've been required to revisit an old zope application built using buildout. The application versions are quite old (3.4.* mostly) and includes old dependencies such as lxml-1.3.1, ZODB3-3.8 amongst others.

As the buildout was running I noticed compile errors and finally it would grind to a halt when import say `_zope_proxy_proxy` which is a compiled .so file.

To cut a long story short the solution for me was to edit `lib/python2.6/config/Makefile` (naturally this is within a `virtualenv`) and change the lines:

CC= /usr/bin/gcc-4.2
CXX= /usr/bin/g++-4.2

to read:

CC= /usr/bin/gcc-4.0
CXX= /usr/bin/g++-4.0

Repoze BFG on Google App Engine

2010-01-09T12:36:00.000-08:00

Introduction

29/02/2010: I've gone a different direction than what is detailed here, instead I am using the approach taken by the bridal-demo. It wasn't too much trouble to plug a repoze.bfg application into the demo code. Hopefully I will write up a post about how I did that in the near future.

I've come back to a google appengine application that I'd put aside for a while. I've been using repoze.bfg in other work so was determined to do the same here. There is a good tutorial for getting a repoze.bfg application running on gae using appengine-monkey which sets up a virtualenv within which the application can be developed. Great, I'm a fan of virtualenv. A point to remember is that dev_appserver cannot be run with the `virtualev`/bin/python. I would forget that; so now I use a `start.sh` script to run the application.

start.sh:

#!/bin/bash
PYTHON=/opt/local/bin/python2.5
GAE_PATH=/usr/local/google_appengine
$PYTHON $GAE_PATH/dev_appserver.py ./

Testing

I found it surprisingly difficult to set up a test environment for gae. My requirements were firstly to be able to use doctests, secondly to have coverage. Coverage comes with nose to I generally use nose for testing. Initially it seemed that `virtualenv`/bin/easy_install nose coverage would do the job until I came to testing models. No google.appengine, no datastore. NoseGAE seemed to be a good answer but wouldn't work well within the virtualenv (again import problems). I also looked at gaeunit but I can't see any benefits to running tests through the browser during development (I rarely even look at a web application that I'm developing - that's what tests are for, to save me the trouble).

However code from gaeunit helped me a lot from which, along with this post, I developed my own solution.

First part of the problem was to get sys.path set up correctly. I took what I needed from runner.py and dev_appserver.py to get the sys.path in working order.

Secondly, I needed to have a test datastore available for testing models, I took a stanza from Dom's post (also used by gaeunit).

Thirdly, when I put() a model an error would be raised about a lack of `app_id` so I solved that with a stanza from $GAE_PATH/tools/dev_appserver.py to read app.yaml and set os.environ['APPLICATION_ID'].

The resulting files then are

test.sh

#!/bin/bash
PYTHON=/opt/local/bin/python2.5
$PYTHON testrunner.py $@

testrunner.py

#!/usr/bin/env python

# a special test runner
import sys
import os

# {{ the first stanza copied from $GAE_PATH/dev_appserver.py
GAE_PATH = "/usr/local/google_appengine"
DIR_PATH = GAE_PATH
SCRIPT_DIR = os.path.join(DIR_PATH, 'google', 'appengine', 'tools')
EXTRA_PATHS = [
    DIR_PATH,
    os.path.join(DIR_PATH, 'lib', 'antlr3'),
    os.path.join(DIR_PATH, 'lib', 'django'),
    os.path.join(DIR_PATH, 'lib', 'webob'),
    os.path.join(DIR_PATH, 'lib', 'yaml', 'lib'),
  ]
sys.path = EXTRA_PATHS + sys.path
# }}

# {{ and the second from runner.py
here = os.path.dirname(__file__)
fixpaths = os.path.join(here, 'fixpaths.py')
execfile(fixpaths)
# }}

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub
from google.appengine.tools.dev_appserver import ReadAppConfig

# read application config file 
appinfo_path = os.path.join(here, 'app.yaml')
config = ReadAppConfig(appinfo_path)

# set application id so that objects can be stored in test database
os.environ['APPLICATION_ID'] = config.application

def main(module=None):
    original_apiproxy = apiproxy_stub_map.apiproxy
    try:
        apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
        temp_stub = datastore_file_stub.DatastoreFileStub('TestDataStore', None, None, trusted=True)
        apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_stub)
        # Allow the other services to be used as-is for tests.
        for name in ['user', 'urlfetch', 'mail', 'memcache', 'images']:
             apiproxy_stub_map.apiproxy.RegisterStub(name, original_apiproxy.GetStub(name))
        from nose.core import TestProgram
        testprogram = TestProgram(module=module)
    except:
        apiproxy_stub_map.apiproxy = original_apiproxy

# accept module name to pass to test runner
try:
    module = sys.argv[1]
except:
    module = None

if __name__ == "__main__":
    main(module=module)

And for completeness here is setup.cfg

[nosetests]
verbose=1
nocapture=1
with-coverage=1
cover-erase=1
cover-inclusive=1
cover-package=mybfgapp

So far, so good. I do rather suspect that this remains incomplete.

References:

http://docs.repoze.org/bfg/1.2/tutorials/gae/index.html
http://blog.appenginefan.com/2008/06/unit-tests-for-google-app-engine-apps.html
http://www.cuberick.com/2008/11/unit-test-your-google-app-engine-models.html
http://code.google.com/p/nose-gae/
http://code.google.com/p/gaeunit/
http://domderrien.blogspot.com/2009/01/automatic-testing-of-gae-applications.html

mod_wsgi on OSX

2009-12-09T17:51:00.000-08:00

The problem: mod_wsgi built fine but:

$ apachectl configtest
httpd: Syntax error on line 118 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_wsgi.so into server: dlopen(/usr/libexec/apache2/mod_wsgi.so, 10
): Symbol not found: _PyExc_RuntimeError\n Referenced from: /usr/libexec/apache2/mod_wsgi.so\n Expected in: dynamic lookup\n

After following along with InstallationOnMacOSX I latterly found this comment on his own post by Tom Giddens to get mod_wsgi to work with apache. I actually went with this method to achieve the same goal. Thanks all.

Some python notes on Leopard 10.5

2009-08-10T03:55:00.000-07:00

Thanks again to Ian Bicking:

lxml: an underappreciated web scraping library

$ STATIC_DEPS=true easy_install 'lxml>=2.2alpha1'

Truly most appreciated.

A Life Stream

2009-04-19T22:54:00.000-07:00

Version II of my life stream effort is acting now as my home page. The first version was using FriendFeed but in this version my server side code is directly requesting the feeds from google, twitter and delicious.

The earlier version used a jsonrpc library but I'm now using web.py to provide the services.

Compiling mod_python on OSX

2009-04-02T12:28:00.000-07:00

Source distribution of mod_python (3.3.1) didn't compile on my iMac, but that from the repository did:

svn co https://svn.apache.org/repos/asf/quetzalcoatl/mod_python/trunk mod_python_src
cd mod_python_src
./configure --with-apxs=/usr/local/apache2/bin/apxs --with-python=/usr/local/Python/Python-2.5.2/bin/python2.5
make
sudo make install

Note that I have built both apache2 and python2.5 from source on my system.

apache, php, postgresql on osx

2008-10-06T11:15:00.000-07:00

I've added some scripts to build (cmmi) software on my mac. Although the software is readily available in binary form or using macports I prefer complete control.

Events app

2008-09-15T03:02:00.000-07:00

A little app using google data calendar api: http://transitiontownthames.org.nz/events.html. At the moment anyone with a google account can add events and therefore I had to disable the 'edit entry' and 'delete entry' functionality. Need to get my head around the authentication particulary access control. I spent around 30 hours coding the thing (including learning from scratch the necessary apis).

Compiling Python2.5 on Leopard

2008-09-05T18:32:00.000-07:00

I had some bother with compiling python2.5 on Leopard. But with the help of Andreas Jung I finally got it to work as I'd like. For what it's worth I've created a shell script for the process. It also includes installing easy_install, lxml and mx packages. (There was a problem with mxExperimental which is fixed with the development snapshot - thanks Marc-Andre.)

First Post

2008-08-05T04:13:00.000-07:00

I suppose one has to start somewhere.

I've created a buildout to get started with gdata.py. I needed to hack the gdata.py package setup.py to use setuptools instead of disutils. Maybe later I'll find a different way to install gdata.py into the buildout but for now I've gone with unpacking the tar into my buildout directory and building the package as a develop egg.

I then created a new tfws.gdata package (for want of a better name) to start playing around with the gdata.py api. The tfws namespace is one under which I've been building python packages in the past (almost exclusively unfinished and unreleased zope work). The new tfws.gdata package today contains only a doctest to ensure that the packages have installed correctly by simple imports.

Then, thanks to wu, I found out why my macports installed python2.5 didn't have ssl or indeed readline support.