Thursday, May 3, 2012

Testing mail server with telnet

POP3

Connect to the server.
telnet mailserver.com pop3
Trying xxx.xxx.xxx.xxx...
Connected to mailserver.com.
Escape character is '^]'.
+OK Dovecot ready.
Login.

user me
+OK
pass mypassword
+OK Logged in.
list
+OK 1 messages:
1 855
retr 1
...

SMTP

Create encoded login details.
perl -MMIME::Base64 -e 'print encode_base64("username");'
perl -MMIME::Base64 -e 'print encode_base64("password");'
Contact server and login.
telnet mailserver.com smtp
EHLO mailserver.com
AUTH LOGIN
The server will ask for username.
334 VXNlcm5hbWU6
Enter the encoded username.
bWU6==
Similarly with password.
334 UGFzc3dvcmQ6
bWUxc3Q=
If all well then server returns.
235 2.0.0 Authentication successful
Cool, now send an email.
MAIL from:me@mailserver.com
250 2.1.0 Ok
RCPT to:them@gmail.com
250 2.1.5 Ok
DATA
354 End data with .
From: me@mailserver.com
To: them@gmail.com
Subject: Test Message

This is a test message.
.
250 2.0.0 Ok: queued as C5C2C7C231
Thanks to http://www.ndchost.com/wiki/mail/test-smtp-auth-telnet

Wednesday, April 7, 2010

Running a HTTP Server in a Thread

I'm building a shop using Django and Satchmo and I ran into an interesting problem while building tests for the Paypal payment module.

The user is sent off to Paypal to process the payment and are returned to the success page. Paypal notifies the site of payment through an 'ipn' url on the shop. Faking the call to the ipn url is easy enough in the test:

>>> postdata = dict(payment_status='Completed',
...     invoice=order.id, memo=memo, txn_id='1234',
...     mc_gross=order.total)
>>> response = client.post('/checkout/paypal/ipn/', postdata)

But the problem then arose because the ipn django view then requests verification from paypal:

def confirm_ipn_data(data, PP_URL):
    # data is the form data that was submitted to the IPN URL.

    newparams = {}
    for key in data.keys():
        newparams[key] = data[key]

    newparams['cmd'] = "_notify-validate"
    params = urlencode(newparams)

    req = urllib2.Request(PP_URL)
    req.add_header("Content-type", "application/x-www-form-urlencoded")
    fo = urllib2.urlopen(req, params)

    ret = fo.read()
    if ret == "VERIFIED":
        log.info("PayPal IPN data verification was successful.")
    else:
        log.info("PayPal IPN data verification failed.")
        log.debug("HTTP code %s, response text: '%s'" % (fo.code, ret))
        return False

    return True

The problem here is opening the url using urllib2. I couldn't use the http://testserver/ that is 'running' in the Django test code. So I tried to start up a simple HTTPServer instance but this would lock up the testrunner. Finally then I came up with the following code which does the job nicely:

# simple server to use as fake verification server
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer
from threading import Thread

class PaypalHandler(BaseHTTPRequestHandler):
    """
    Simple server to return verified value for our paypal tests
    """
    def do_POST(self):
        self.send_response(200, 'OK')
        self.send_header('Content-type', 'text/html')
        self.end_headers()
        self.wfile.write( "VERIFIED" )

class RunThread(Thread):
    def __init__(self, port):
        Thread.__init__(self)
        self.server = HTTPServer(('127.0.0.1', port), PaypalHandler)
    def run(self):
        self.server.serve_forever()

server_thread = None

def start_server(port):
    global server_thread
    server_thread = RunThread(port)
    server_thread.start()

def stop_server():
    global server_thread
    server_thread.server.socket.close()

Now in my test setup I can call

def setUp(suite):
    testing.start_server(10080)

def tearDown(suite):
    testing.stop_server()

And it works as expected.

Sunday, February 28, 2010

Snow Leopard and Zope3

I've been required to revisit an old zope application built using buildout. The application versions are quite old (3.4.* mostly) and includes old dependencies such as lxml-1.3.1, ZODB3-3.8 amongst others.

As the buildout was running I noticed compile errors and finally it would grind to a halt when import say `_zope_proxy_proxy` which is a compiled .so file.

To cut a long story short the solution for me was to edit `lib/python2.6/config/Makefile` (naturally this is within a `virtualenv`) and change the lines:

CC= /usr/bin/gcc-4.2
CXX= /usr/bin/g++-4.2

to read:

CC= /usr/bin/gcc-4.0
CXX= /usr/bin/g++-4.0

Saturday, January 9, 2010

Repoze BFG on Google App Engine

Introduction

29/02/2010: I've gone a different direction than what is detailed here, instead I am using the approach taken by the bridal-demo. It wasn't too much trouble to plug a repoze.bfg application into the demo code. Hopefully I will write up a post about how I did that in the near future.
I've come back to a google appengine application that I'd put aside for a while. I've been using repoze.bfg in other work so was determined to do the same here. There is a good tutorial for getting a repoze.bfg application running on gae using appengine-monkey which sets up a virtualenv within which the application can be developed. Great, I'm a fan of virtualenv. A point to remember is that dev_appserver cannot be run with the `virtualev`/bin/python. I would forget that; so now I use a `start.sh` script to run the application.

start.sh:
#!/bin/bash
PYTHON=/opt/local/bin/python2.5
GAE_PATH=/usr/local/google_appengine
$PYTHON $GAE_PATH/dev_appserver.py ./

Testing

I found it surprisingly difficult to set up a test environment for gae. My requirements were firstly to be able to use doctests, secondly to have coverage. Coverage comes with nose to I generally use nose for testing. Initially it seemed that `virtualenv`/bin/easy_install nose coverage would do the job until I came to testing models. No google.appengine, no datastore. NoseGAE seemed to be a good answer but wouldn't work well within the virtualenv (again import problems). I also looked at gaeunit but I can't see any benefits to running tests through the browser during development (I rarely even look at a web application that I'm developing - that's what tests are for, to save me the trouble).

However code from gaeunit helped me a lot from which, along with this post, I developed my own solution.

First part of the problem was to get sys.path set up correctly. I took what I needed from runner.py and dev_appserver.py to get the sys.path in working order.

Secondly, I needed to have a test datastore available for testing models, I took a stanza from Dom's post (also used by gaeunit).

Thirdly, when I  put() a model an error would be raised about a lack of  `app_id` so I solved that with a stanza from $GAE_PATH/tools/dev_appserver.py to read app.yaml and set os.environ['APPLICATION_ID'].

The resulting files then are

test.sh
#!/bin/bash
PYTHON=/opt/local/bin/python2.5
$PYTHON testrunner.py $@

testrunner.py
#!/usr/bin/env python

# a special test runner
import sys
import os

# {{ the first stanza copied from $GAE_PATH/dev_appserver.py
GAE_PATH = "/usr/local/google_appengine"
DIR_PATH = GAE_PATH
SCRIPT_DIR = os.path.join(DIR_PATH, 'google', 'appengine', 'tools')
EXTRA_PATHS = [
    DIR_PATH,
    os.path.join(DIR_PATH, 'lib', 'antlr3'),
    os.path.join(DIR_PATH, 'lib', 'django'),
    os.path.join(DIR_PATH, 'lib', 'webob'),
    os.path.join(DIR_PATH, 'lib', 'yaml', 'lib'),
  ]
sys.path = EXTRA_PATHS + sys.path
# }}

# {{ and the second from runner.py
here = os.path.dirname(__file__)
fixpaths = os.path.join(here, 'fixpaths.py')
execfile(fixpaths)
# }}

from google.appengine.api import apiproxy_stub_map
from google.appengine.api import datastore_file_stub
from google.appengine.tools.dev_appserver import ReadAppConfig

# read application config file 
appinfo_path = os.path.join(here, 'app.yaml')
config = ReadAppConfig(appinfo_path)

# set application id so that objects can be stored in test database
os.environ['APPLICATION_ID'] = config.application

def main(module=None):
    original_apiproxy = apiproxy_stub_map.apiproxy
    try:
        apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
        temp_stub = datastore_file_stub.DatastoreFileStub('TestDataStore', None, None, trusted=True)
        apiproxy_stub_map.apiproxy.RegisterStub('datastore', temp_stub)
        # Allow the other services to be used as-is for tests.
        for name in ['user', 'urlfetch', 'mail', 'memcache', 'images']:
             apiproxy_stub_map.apiproxy.RegisterStub(name, original_apiproxy.GetStub(name))
        from nose.core import TestProgram
        testprogram = TestProgram(module=module)
    except:
        apiproxy_stub_map.apiproxy = original_apiproxy

# accept module name to pass to test runner
try:
    module = sys.argv[1]
except:
    module = None

if __name__ == "__main__":
    main(module=module)

And for completeness here is setup.cfg
[nosetests]
verbose=1
nocapture=1
with-coverage=1
cover-erase=1
cover-inclusive=1
cover-package=mybfgapp

So far, so good. I do rather suspect that this remains incomplete.

References:

http://docs.repoze.org/bfg/1.2/tutorials/gae/index.html
http://blog.appenginefan.com/2008/06/unit-tests-for-google-app-engine-apps.html
http://www.cuberick.com/2008/11/unit-test-your-google-app-engine-models.html
http://code.google.com/p/nose-gae/
http://code.google.com/p/gaeunit/
http://domderrien.blogspot.com/2009/01/automatic-testing-of-gae-applications.html

Wednesday, December 9, 2009

mod_wsgi on OSX

The problem: mod_wsgi built fine but:

$ apachectl configtest
httpd: Syntax error on line 118 of /private/etc/apache2/httpd.conf: Cannot load /usr/libexec/apache2/mod_wsgi.so into server: dlopen(/usr/libexec/apache2/mod_wsgi.so, 10
): Symbol not found: _PyExc_RuntimeError\n Referenced from: /usr/libexec/apache2/mod_wsgi.so\n Expected in: dynamic lookup\n


After following along with InstallationOnMacOSX I latterly found this comment on his own post by Tom Giddens to get mod_wsgi to work with apache. I actually went with this method to achieve the same goal. Thanks all.

Monday, August 10, 2009

Some python notes on Leopard 10.5

Thanks again to Ian Bicking:

lxml: an underappreciated web scraping library

$ STATIC_DEPS=true easy_install 'lxml>=2.2alpha1'

Truly most appreciated.

Sunday, April 19, 2009

A Life Stream

Version II of my life stream effort is acting now as my home page. The first version was using FriendFeed but in this version my server side code is directly requesting the feeds from google, twitter and delicious.

The earlier version used a jsonrpc library but I'm now using web.py to provide the services.