Repoze BFG on Google App Engine

Introduction

29/02/2010: I've gone a different direction than what is detailed here, instead I am using the approach taken by the bridal-demo. It wasn't too much trouble to plug a repoze.bfg application into the demo code. Hopefully I will write up a post about how I did that in the near future.

I've come back to a google appengine application that I'd put aside for a while. I've been using repoze.bfg in other work so was determined to do the same here. There is a good tutorial for getting a repoze.bfg application running on gae using appengine-monkey which sets up a virtualenv within which the application can be developed. Great, I'm a fan of virtualenv. A point to remember is that dev_appserver cannot be run with the `virtualev`/bin/python. I would forget that; so now I use a `start.sh` script to run the application.

start.sh:

#!/bin/bash
PYTHON=/opt/local/bin/python2.5
GAE_PATH=/usr/local/google_appengine
$PYTHON $GAE_PATH/dev_appserver.py ./

Testing

I found it surprisingly difficult to set up a test environment for gae. My requirements were firstly to be able to use doctests, secondly to have coverage. Coverage comes with nose to I generally use nose for testing. Initially it seemed that `virtualenv`/bin/easy_install nose coverage would do the job until I came to testing models. No google.appengine, no datastore. NoseGAE seemed to be a good answer but wouldn't work well within the virtualenv (again import problems). I also looked at gaeunit but I can't see any benefits to running tests through the browser during development (I rarely even look at a web application that I'm developing - that's what tests are for, to save me the trouble).

However code from gaeunit helped me a lot from which, along with this post, I developed my own solution.

First part of the problem was to get sys.path set up correctly. I took what I needed from runner.py and dev_appserver.py to get the sys.path in working order.

Secondly, I needed to have a test datastore available for testing models, I took a stanza from Dom's post (also used by gaeunit).

Thirdly, when I  put() a model an error would be raised about a lack of  `app_id` so I solved that with a stanza from $GAE_PATH/tools/dev_appserver.py to read app.yaml and set os.environ['APPLICATION_ID'].

The resulting files then are

test.sh

#!/bin/bash
PYTHON=/opt/local/bin/python2.5
$PYTHON testrunner.py $@

testrunner.py

#!/usr/bin/env python

# a special test runner
import sys
import os

# {{ the first stanza copied from $GAE_PATH/dev_appserver.py
GAE_PATH = "/usr/local/google_appengine"
DIR_PATH = GAE_PATH
SCRIPT_DIR = os.path.join(DIR_PATH, 'google', 'appengine', 'tools')
EXTRA_PATHS = [
    DIR_PATH,
    os.path.join(DIR_PATH, 'lib', 'antlr3'),
    os.path.join(DIR_PATH, 'lib', 'django'),
    os.path.join(DIR_PATH, 'lib', 'webob'),
    os.path.join(DIR_PATH, 'lib', 'yaml', 'lib'),
  ]
sys.path = EXTRA_PATHS + sys.path
# }}

# {{ and the second from runner.py
here = os.path.dirname(__file__)
fixpaths = os.path.join(here, 'fixpaths.py')
execfile(fixpaths)
# }}

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub
from google.appengine.tools.dev_appserver import ReadAppConfig

# read application config file 
appinfo_path = os.path.join(here, 'app.yaml')
config = ReadAppConfig(appinfo_path)

# set application id so that objects can be stored in test database
os.environ['APPLICATION_ID'] = config.application

def main(module=None):
    original_apiproxy = apiproxy_stub_map.apiproxy
    try:
        apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
        temp_stub = datastore_file_stub.DatastoreFileStub('TestDataStore', None, None, trusted=True)
        apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_stub)
        # Allow the other services to be used as-is for tests.
        for name in ['user', 'urlfetch', 'mail', 'memcache', 'images']:
             apiproxy_stub_map.apiproxy.RegisterStub(name, original_apiproxy.GetStub(name))
        from nose.core import TestProgram
        testprogram = TestProgram(module=module)
    except:
        apiproxy_stub_map.apiproxy = original_apiproxy

# accept module name to pass to test runner
try:
    module = sys.argv[1]
except:
    module = None

if __name__ == "__main__":
    main(module=module)

And for completeness here is setup.cfg

[nosetests]
verbose=1
nocapture=1
with-coverage=1
cover-erase=1
cover-inclusive=1
cover-package=mybfgapp

So far, so good. I do rather suspect that this remains incomplete.

References:

http://docs.repoze.org/bfg/1.2/tutorials/gae/index.html
http://blog.appenginefan.com/2008/06/unit-tests-for-google-app-engine-apps.html
http://www.cuberick.com/2008/11/unit-test-your-google-app-engine-models.html
http://code.google.com/p/nose-gae/
http://code.google.com/p/gaeunit/
http://domderrien.blogspot.com/2009/01/automatic-testing-of-gae-applications.html