Saturday, January 9, 2010

Repoze BFG on Google App Engine


29/02/2010: I've gone a different direction than what is detailed here, instead I am using the approach taken by the bridal-demo. It wasn't too much trouble to plug a repoze.bfg application into the demo code. Hopefully I will write up a post about how I did that in the near future.
I've come back to a google appengine application that I'd put aside for a while. I've been using repoze.bfg in other work so was determined to do the same here. There is a good tutorial for getting a repoze.bfg application running on gae using appengine-monkey which sets up a virtualenv within which the application can be developed. Great, I'm a fan of virtualenv. A point to remember is that dev_appserver cannot be run with the `virtualev`/bin/python. I would forget that; so now I use a `` script to run the application.


I found it surprisingly difficult to set up a test environment for gae. My requirements were firstly to be able to use doctests, secondly to have coverage. Coverage comes with nose to I generally use nose for testing. Initially it seemed that `virtualenv`/bin/easy_install nose coverage would do the job until I came to testing models. No google.appengine, no datastore. NoseGAE seemed to be a good answer but wouldn't work well within the virtualenv (again import problems). I also looked at gaeunit but I can't see any benefits to running tests through the browser during development (I rarely even look at a web application that I'm developing - that's what tests are for, to save me the trouble).

However code from gaeunit helped me a lot from which, along with this post, I developed my own solution.

First part of the problem was to get sys.path set up correctly. I took what I needed from and to get the sys.path in working order.

Secondly, I needed to have a test datastore available for testing models, I took a stanza from Dom's post (also used by gaeunit).

Thirdly, when I  put() a model an error would be raised about a lack of  `app_id` so I solved that with a stanza from $GAE_PATH/tools/ to read app.yaml and set os.environ['APPLICATION_ID'].

The resulting files then are
#!/usr/bin/env python

# a special test runner
import sys
import os

# {{ the first stanza copied from $GAE_PATH/
GAE_PATH = "/usr/local/google_appengine"
SCRIPT_DIR = os.path.join(DIR_PATH, 'google', 'appengine', 'tools')
    os.path.join(DIR_PATH, 'lib', 'antlr3'),
    os.path.join(DIR_PATH, 'lib', 'django'),
    os.path.join(DIR_PATH, 'lib', 'webob'),
    os.path.join(DIR_PATH, 'lib', 'yaml', 'lib'),
sys.path = EXTRA_PATHS + sys.path
# }}

# {{ and the second from
here = os.path.dirname(__file__)
fixpaths = os.path.join(here, '')
# }}

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub
from import ReadAppConfig

# read application config file 
appinfo_path = os.path.join(here, 'app.yaml')
config = ReadAppConfig(appinfo_path)

# set application id so that objects can be stored in test database
os.environ['APPLICATION_ID'] = config.application

def main(module=None):
    original_apiproxy = apiproxy_stub_map.apiproxy
        apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
        temp_stub = datastore_file_stub.DatastoreFileStub('TestDataStore', None, None, trusted=True)
        apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_stub)
        # Allow the other services to be used as-is for tests.
        for name in ['user', 'urlfetch', 'mail', 'memcache', 'images']:
             apiproxy_stub_map.apiproxy.RegisterStub(name, original_apiproxy.GetStub(name))
        from nose.core import TestProgram
        testprogram = TestProgram(module=module)
        apiproxy_stub_map.apiproxy = original_apiproxy

# accept module name to pass to test runner
    module = sys.argv[1]
    module = None

if __name__ == "__main__":

And for completeness here is setup.cfg

So far, so good. I do rather suspect that this remains incomplete.


1 comment:

Chris said...

Very interesting. I keep falling back to gaeunit on projects because it is just so easy to drop in to a new project and validate all the models and logic. I run into the limitation when I start wanting code coverage and a few other things. I've tried twice to get app-engine-patch, django, nose, and nose-gae all working nicely together but each time a few small versioning-related issues have turned the effort into a time sink. So I keep falling back to simple gaeunit and living with what I can get there.

Thanks for the detailed write-up. This almost looks simple enough to give nose another try.