An Experiment In Scotch

I write to discover what I believe

Tag: python

Installing Django

This is just a page for me to remember what I did while installing Django so that when something breaks in six months, I have a point of reference.

Followed this page: Setup Django on Lion to some degree. Worked through the tutorial on djangoproject as well.

Made changes to where site packages are created, namely ~/.local/lib/python2.7 instead of the default.

When installing virtualenv and virtualenvwrapper, I received the following error related to a virtualenvwrapper dependency on stevedore: “raise TypeError, “dist must be a Distribution instance”
TypeError: dist must be a Distribution instance”. I fixed this by installing virtualenv, virtualenvwrapper and stevedore individually instead of with the command in the installation guide linked above.

Virtualenv sites exist in /Users//Sites/env.

Word Wrap Kata in Python

Last month at Dallas Hack Club, we did the Word Wrap Kata from Uncle Bob Martin’s “Clean Coder”. I got there a touch late and rapidly figured out that we were doing it in Ruby. Now, my Ruby skills are right up there with my Mandarin Chinese skills which is to say, I can’t even order a scotch or find the restroom so I quickly figured out that if I followed along, I’d be sitting in a puddle of pee wishing I was drunk. So I made the executive decision to ignore most everyone else and do it in parallel with the group but in Python since I can at least code like I’m drunk in Python.

My experience was surprisingly similar to the post linked above, e.g. I went down the hardest path trying to solve the wrong test. I’ve since gone back and deleted the code but the pseudocode went something like this:
[sourcecode language=”python”]Break text into a list using a list comprehension based on the length of the column
Start looking for words with spaces
Try to repiece things together based on the column length and the last space
Cry
[/sourcecode]

Clearly, this wasn’t going to work. About that time, we ran out of time and went to the Flying Saucer to drink beer and pretend like we weren’t geeks. But I decided to finish the kata this week during a slow period at work. Since I didn’t have Jerry there to talk me through how dumb I was, I started reading the post up until the point where it describes writing the wrong test and spending too much time trying to solve it. So I backed up, deleted everything and started down the easier path of solving all the non-space related issues first. Once I did that, and once I figured out that recursion was going to be a big help, the project really got easier.

The final solution is below:
[sourcecode language=”python”]
def wrap(text, num):

if len(text) <= num:
return text
elif text[num-1] == ‘ ‘:
return text[:num].rstrip() + ‘rn’ + wrap(text[num:], num)
elif text[:num].find(‘ ‘) > -1:
rightSpaceIdx = text.rfind(‘ ‘)
return text[:rightSpaceIdx] + ‘rn’ + wrap(text[rightSpaceIdx:].lstrip(‘ ‘), num)
else:
return text[:num] + ‘rn’ + wrap(text[num:].lstrip(), num)
[/sourcecode]

It’s fascinating how quickly the algorithm starts to come together when you write the correct test. The problem is, I’m terrible at all this so it’s going to take some time getting enough experience to correctly select what test to write next. I was writing a ton of code trying to solve a problem that was too hard and the solution I eventually would have come up with would have been effective but brittle.

The more I do TDD/BDD, the more I realize that it is *THE* way to develop software, especially if you’re working in a dynamic language. I’m currently working through the tutorials at Ruby Tutorials and it’s great to see that TDD is a fundamental part of the process. As I learned Python and Pylons, it was up to me to figure out best practices it seemed like and that’s workable but frustrating in the long run. I’m planning on doing more katas in an effort to improve my craft.

Route Gotchas In Pylons

I’ve been working on a Pylons app quite a bit lately and occasionally I run across issues that warrant documentation on the interwebs. That happened today concerning routes and how Pylons deals with them.

If you start getting an Import error that says “No module named content found”, you’ve run into it. According to the Pylons book, “Routes has a surprising legacy feature that means that if you don’t specify a controller and an action for a particular route, the implicit defaults of controller=’content’ and action=’index’ will be used for you”. This is certainly surprising to me and as always, things that are implicit tend to annoy me. Luckily, you can change this behavior. In your config/routing.py file, set map.explicit = True and then you’ll need to alter the route that is giving you problems to make the controller and action explicit. For example:

Before
[sourcecode language=”python”]def make_map(config):
"""Create, configure and return the routes Mapper"""
map = Mapper(directory=config[‘pylons.paths’][‘controllers’],
always_scan=config[‘debug’])
map.minimization = False
map.explicit = False

# The ErrorController route (handles 404/500 error pages); it should
# likely stay at the top, ensuring it can always be resolved
map.connect(‘/error/{action}’, controller=’error’)
map.connect(‘/error/{action}/{id}’, controller=’error’)

# CUSTOM ROUTES HERE
map.connect(‘schedule_entry’, ‘schedule/index/{scheduleDay}’)
map.connect(‘/{controller}/{action}’)
map.connect(‘/{controller}/{action}/{id}’)

return map
[/sourcecode]

As you can see, the default behavior is for map.explicit to be set to False. The first route under CUSTOM ROUTES HERE is a named route that to my eye should match schedule up with a controller and index up with the action. Unfortunately, instead it tries to find the default implicit controller “content” which it can’t find and that throws the ImportError listed above. To fix it do this:

After
[sourcecode language=”python”]
def make_map(config):
"""Create, configure and return the routes Mapper"""
map = Mapper(directory=config[‘pylons.paths’][‘controllers’],
always_scan=config[‘debug’])
map.minimization = False
map.explicit = True

# The ErrorController route (handles 404/500 error pages); it should
# likely stay at the top, ensuring it can always be resolved
map.connect(‘/error/{action}’, controller=’error’)
map.connect(‘/error/{action}/{id}’, controller=’error’)

# CUSTOM ROUTES HERE
map.connect(‘/schedule/index/{scheduleDay}’, controller=’schedule’, action=’index’)
map.connect(‘/{controller}/{action}’)
map.connect(‘/{controller}/{action}/{id}’)

return map
[/sourcecode]

Now the map is set to explicit and the controller and action are explicitly specified which works just fine. This all may be an artifact my novice understanding of Routes but since the book documents it this way, I guess this is the way I’m going to do it.

TDD with Pylons

I’ve written about test driven development and Pylons before but there have apparently been some changes to how it all works since Pylons 1.0. I didn’t run across anything in the documentation detailing the changes, specifically to the template context global and how you access it in your tests.

From the Pylons docs on testing:

Pylons will provide several additional attributes for the paste.fixture response object that let you access various objects that were created during the web request:

  1. session — Session object
  2. req — Request object
  3. c — Object containing variables passed to templates
  4. g — Globals object

To use them, merely access the attributes of the response after you’ve used a get/post command:
[sourcecode language=”python”]response = app.get(‘/some/url’)
assert response.session[‘var’] == 4
assert ‘REQUEST_METHOD’ in response.req.environ[/sourcecode]

As it turns out, that’s sort of true. I have a test that looks like this:

[sourcecode language=”python”]from nbapowerrank.tests import *

class TestGamedetailsController(TestController):

def test_index(self):
response = self.app.get(url(controller=’admin/gamedetails’, action=’index’))
self.assertEqual(len(response.c.games) > 0, True)[/sourcecode]

and a controller that looks like this:

[sourcecode language=”python”]from nbapowerrank.lib.base import BaseController, render
from nbapowerrank.model import meta
from nbapowerrank.model.game import Game

log = logging.getLogger(__name__)

class GamedetailsController(BaseController):
def __before__(self):
self.game_q = meta.Session.query(Game)

def index(self):
yesterday = datetime.today() – timedelta(1)
c.games = self.game_q.filter_by(gamedate=yesterday).all()

return render(‘/gamedetails.mako’)[/sourcecode]

Unfortunately, contra the documentation that says the “c” alias will be available on the test response object, that test always fails with an AttributeError stating that in fact, the c attribute does not exist. It frustrated me even more because all the other attributes that are supposed to be on the response like session and g were in fact there. After doing some random digging, I came across this in the Pylons 1.0 roadmap: “Deprecate pylons.c, pylons.g, and pylons.buffet. These have been disrecommended since 0.9.7.”

Apparently, they are/have deprecated using the c alias for tmpl_context even though when you create a controller under Pylons 1.0, it still aliases the context as c. Sigh. So in order to test data that you have added to your context for templating, your test should use the explicit tmpl_context instead of the c like this:

[sourcecode language=”python”]from nbapowerrank.tests import *

class TestGamedetailsController(TestController):

def test_index(self):
response = self.app.get(url(controller=’admin/gamedetails’, action=’index’))
self.assertEqual(len(response.tmpl_context.games) > 0, True)[/sourcecode]

And then all will be well with your world. I do have to say that compared to both ASP.Net MVC and Rails, the testing support in Python/pylons seems to be a second class citizen. I’m not sure why that is because as a dynamic language, it seems to benefit greatly from a TDD approach. Maybe it’s just me getting back into a framework after a year’s worth of changes.

Anyway, that may help some people out there trying to search in the googleplex about TDD and Pylons.

Python Development on Windows

I’ve been rededicating myself to a Python project over the last couple of days and hence have been trying to get my Dell Windows 7 laptop set up for Python development again. It hasn’t been particularly easy. The things I’ve run into so far:

1. Python 2.6 happily runs on Windows 64bit whatever but not all 3rd party libraries are so forward thinking. Specifically, if you’ve got a project using PostgreSQL and SqlAlchemy, SqlAlchemy will complain about the Python driver psycopg2 on 64bit systems. There are other drivers out there but my solution was just to rollback my 64bit installation of Python and reinstall a 32bit version. That seems to be working fine though I haven’t gotten to the point of actually reading and writing data yet.

2. PYTHONPATH on Windows via the registry doesn’t seem to work, at least on my Windows 7 64bit installation. I have added the required registry keys per the documentation, rebooted 42 times and done the Guido van Rossum Purple Rain Remix dance with exactly zero results. For now, I’m just creating an environment variable the old fashioned way.

3. The real problem that led me to having to modify PYTHONPATH was that package imports just don’t seem to be working. I’m sure this is me being the Python equivalent of Charlie in Flowers For Algernon. I think in the past I must have had some magical setting or batch file that modified PYTHONPATH at startup or something but I wish my subpackage imports worked without having to resort to that.

On the plus side, my first TDD test passed first time up so I’ve got that going for me, which is nice. At least the code isn’t so stale that it doesn’t even function anymore.

Making It Easy To Learn Programming

Glenn Reynolds talks about a shortage of geeks and one of his readers comments that a key contribution to this is when computers stopped shipping with BASIC. Glenn makes the plea to computer makers to include BASIC on the computers as a public service. Of course, Macs already ship with both Python and Ruby, surely superior languages to BASIC and excellent learning languages as well.

On a Mac, all you have to do is fire up a terminal (Apple Key + Space Bar and then type “Terminal”), type “python” in the new terminal and you have a place to start learning programming. Work through the Python tutorial and you’re well on the way to becoming a novice Python programmer.

Of course, on a Windows machine you’re going to have to do a little more work because, well, it is Windows after all.

TDD With Python and Pylons

I’ve been doing some development on a website using Python and the Pylons web framework. I’m trying to stay pretty strict with Test Driven Development (TDD) though I run into problems because I’m still a complete novice with Pylons and a half-complete novice with Python. In my experience so far, unit tests are moderately difficult in Pylons and turn out to be something closer to the bastard stepchild of a redneck unit test and a 5th Avenue socialite mom. I feel that way because the unit tests require the Pylons framework to be set up correctly. They also often go outside their boundaries and I haven’t looked into any mock frameworks, though I have the feeling that using a mock framework with a dynamic language like Python is probably a stupid thing to say in public.

Regardless, I have really started to enjoy working with Pylons and that stems from the actual functional tests that are available through the framework. Specifically, my development flow has been something like this:

  • Write new functional test of the web site
  • Run tests to see them error out. Typical error message is that an action isn’t implemented on the controller.
  • Implement the basic controller and the action but leave out the functionality under test.
  • Rerun the test to see it fail.
  • Implement the functionality necessary to get a passing test. This often includes implementing database tables, keys, getting SqlAlchemy set up to correctly map data to objects and creating new templates for HTML.

The functional tests are nice because while they aren’t confirming look and feel type stuff, they at least put the flow of the application under test. I’m a purist when it comes to having unit tests only test the code they are intended for but I’m not a purist when it comes to writing unit tests first as the only way to design the application. With a website like this, I’m pretty happy writing functional tests to drive out the design of the web site.

Here are some code snippets from my current website (try to ignore the fact that this looks like it might have something to do with horses and the Kentucky Derby, especially if you work for the NSA.)

First a test:

from darlydowns.tests import *
from darlydowns.model import meta
from darlydowns.model import horse


class TestHorseController(TestController):
    def test_index(self):
        # setting up a temp horse to make sure one exists for the test
        tempHorse = horse.Horse('my temp', 'my description')
        meta.Session.save(tempHorse)
        meta.Session.commit()

        response = self.app.get(url_for(controller='horse'))
        ## Test response...
        assert len(response.c.horses) > 0

        meta.Session.delete(tempHorse)
        meta.Session.commit()

Here we have a test that tests the response from a request for a URL that is handled by the controller “horse”. This controller grabs all the horses in the database and displays them in a table. The test saves a temp horse, gets the list, verifies the list contains at least 1 horse and then deletes the temp horse to clean up after itself.

Here’s the controller code that allows the test to pass:


import logging

from darlydowns.model import meta
from darlydowns.lib.base import *
from darlydowns.model import horse
from darlydowns.model.horse import Horse

log = logging.getLogger(__name__)

class HorseController(BaseController):

    def index(self):
        c.horses = [horse for horse in meta.Session.query(Horse).all()]
        return render('/horses.mako')

One of the beauties of Pylons is how little code is required to do something, once the project is set up. Here our HorseController has an action of “index” which is defined as a method. It grabs all the horses from the database and then uses a list comprehension to collect them into the c.horses variable. It then renders the template “horses.mako” which knows how to layout the web page using the horses found in the database.

Once this was done, I wrote code to save a horse, all driven out by the functional tests. I’ve been pretty happy with how the design is driven from these tests as it’s often quite clear where to proceed next in the application from the last test. Lots of times, other functionality comes up that doesn’t logically flow next but I just add that to a growing todo list in the project to make sure nothing is skipped.

Pylons takes a little getting used to, especially since I come from a static, everything in once solution sort of background. With Pylons, you need to learn Routes and Mako and SqlAlchemy but once you get your head around all those tools, it’s a joy to work with.

UPDATE: Welcome to everyone coming here from the Python subreddit. If you have any tips for testing using Pylons, feel free to drop me a comment. I’d love to hear about other people’s experiences.