Greg Allard

I found
one library that was linked everywhere, but it wasn’t working for me. I was always getting 400 Bad Request when using it. Chris Conover was able to get the following code working.

import urllib2
 
protocol = 'https://'
url = 'example.com'
command = '/projects.xml'
headers = {'Accept' : 'application/xml', 
'Content-type' : 'applications/xml'}
username = 'x'
password = 'y'
 
# Setup password stuff
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
authhandler = urllib2.HTTPBasicAuthHandler(passman)
opener = urllib2.build_opener(authhandler)
urllib2.install_opener(opener)
 
# Send the request and get response
req = urllib2.Request(protocol + url + command, None, headers)
response = urllib2.urlopen(req)
results = response.read()
 
print results

I thought it was a problem with how the authorization was formed so based on the above code I modified the old basecamp.py file and I was able to get a response. The following is what I changed.

Around line 64

    def __init__(self, username, password, protocol, url):
        self.baseURL = protocol+url
        if self.baseURL[-1] == '/':
            self.baseURL = self.baseURL[:-1]
 
        passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
        passman.add_password(None, url, username, password)
        authhandler = urllib2.HTTPBasicAuthHandler(passman)
 
        self.opener = urllib2.build_opener(authhandler)

And around line 142

path = '/projects.xml'

With that I was able to use basecamp.py to retrieve a list of projects. Other modifications may be needed for other features, but that was all I planned on using.

Here is an example of using
ElementTree to parse the XML response to get the names of all of the projects returned from basecamp.

import elementtree.ElementTree as ET
from basecamp import Basecamp
 
protocol = 'https://'
url = 'example.com'
username = 'x'
password = 'y'
 
bc = Basecamp(username, password, protocol, url)
projects = bc.projects()
tree = ET.fromstring(projects)
tags = tree.getiterator(tag='project')
 
for t in tags:
    project_name = t.findtext('name')

Related posts:

1. Python Projects in Users’ Home Directories with wsgi Letting users put static files and php files in a…
2. How to Add Locations to Python Path for Reusable Django Apps In my previous post I talk about reusable apps, but…
3. Setting up Apache2, mod_python, MySQL, and Django on Debian Lenny or Ubuntu Hardy Heron Both Debian and Ubuntu make it really simple to get…

The idea for this site came when some co-workers and I were collecting our W2’s and found letters attached with some HR information. One of them had a ridiculously long URL at the bottom of it (…Enroll%20in%20UCF%20Retirement%20Plans…). Before seeing that URL I hadn’t thought of the convenience of services like TinyURL outside of the internet. We realized it would be simple enough to write one that is specific to UCF.

We decided on a few simple features and jumped into pair programming the site with django.  Since Jason was new to Django, he obvserved while I drove. It took less than the rest of the day to finish the site.

Features

Only allows domains that we specify

We created a custom Form Field to accomplish this and made it able to accept a tuple of allowed domains. If anyone needs this on their site they can use the following code.

from django import forms
 
class URLDomainField(forms.URLField):
    domain = ''
 
    def __init__(self, *args, **kwargs):
        # set domain to passed value or default
        self.domain = kwargs.get('domain', ('gregallard.com', 'isthemarketdown.com', 'codespatter.com'))
 
        # remove from list if exists
        try:
            del kwargs['domain']
        except:
            pass
 
        # call parent init
        super(URLDomainField, self).__init__(*args, **kwargs)
 
    def clean(self, value):
        # call parent clean
        value = super(URLDomainField, self).clean(value)
 
        from urlparse import urlparse
        o = urlparse(value)
 
        # endswith accepts tuples and will try them all and will return false if none match
        if not o.hostname.endswith(self.domain):
            raise forms.ValidationError('%s is not a valid url! %s domains only.' % (value, self.domain))
 
        return value

The code to use this would look like this:

class LinkForm(forms.Form):
    url   = URLDomainField(domain=('ivylees.com', 'ucf.edu'))

Automatically creates 5 character alphanumeric string

This method in the model creates a string and makes sure it isn’t in use yet:

    def make_short(self):
        from random import Random
        import string
        cool = False
        while not cool:
            self.short = ''.join( Random().sample(string.letters+string.digits, 5) )
            try:
                r = Link.objects.get(short=self.short)
            except Link.DoesNotExist:
                if self.short != "admin" and self.short != "thank":
                    cool = True

Allows for custom strings

By default it will create a 5 character alphanumeric string to go at the end of the URL, however we added a form field to allow users to specify their own string so that the URL might have more meaning. To strip non alphanumeric characters, we created a simple clean method in the model:

    def clean_short(self):
        import re
        # ^ as first character inside [] negates the set
        # find everything that isn't alphanumeric or a -
        self.short = re.sub('[^\w|\-]', '_', self.short)

Won’t create more short links

If a URL has been submitted before, the site will not create an extra URL for it, instead it will return the existing one to the user. To do this, we added some functionality to the save method:

    def save(self, **kwargs):
        link = Link.objects.filter(url=self.url)[:1]
 
        # if one exists, return it, otherwise save it
        if link:
            # there should be a better way to do this
            # but self = link doesn't work
            self.url   = link[0].url
            self.short = link[0].short
            self.created = link[0].created
            self.id = link[0].id
        else:
            if self.short == '':
                self.make_short()
            else:
                self.clean_short()
            super(Link, self).save(**kwargs)

Just a Prototype

We just wanted to create something simple as a prototype so that hopefully some of the higher-ups will like the idea and we can put it into production.

Pair Programming

This was the first time I had any experience with pair programming and I definitely think it’s a great idea. Jason learned a lot about django, caught my mistakes, and pointed out other things. I solidified my knowledge by showing him what I knew and we both learned some valuable things. For example: using print foo will be displayed in the command window when you are using the django development server. I foresee more pair programming in my future.

Update 2009-01-28

Tim recommended that I remove the chance of profanity to be automatically generated for the url and suggested removing all vowels so that no words will be there. This is the line I added to achieve that.

letters = re.sub('a|e|i|o|u|A|E|I|O|U', '', string.letters)

Template tags can be useful for making your applications more reusable by other projects. For this example I will be adding to the
books project that I started in a previous post. Also, I’ve bundled the
example files into a google code project.

Start off by creating a folder called templatetags in your app directory and create two files in it. The first one named __init__.py and the second book_tags.py. There’s 3 things that we need to accomplish with our template tags. The first is to create a tag that will output the url for the action of the form. For example, {% get_book_form_url foo_object %}Next we need to get the form and assign it to a template variable that can be specified by the template variable. For example, {% book_form as bar_var %}. And the third template tag will get the books for an object and place in a template variable. For example, {% books_for_object foo_object as bar_var %}.

from django.template import Library, Node, TemplateSyntaxError
from django.template import Variable, resolve_variable
from django.utils.translation import ugettext as _
from django.contrib.contenttypes.models import ContentType
from django.core.urlresolvers import reverse
from books.models import Book
 
register = Library()
 
def get_contenttype_kwargs(content_object):
    """
    Gets the basic kwargs necessary for form submission url
    """
    kwargs = {'content_type_id':
        ContentType.objects.get_for_model(content_object).id,
    'object_id':
        getattr(content_object, 'pk',
            getattr(content_object, 'id')),
    }
    return kwargs
 
def get_book_form_url(content_object):
    """
    prints url for form action
    """
    kwargs = get_contenttype_kwargs(content_object)
    return reverse('new_book', kwargs=kwargs)
 
class BooksForObjectsNode(Node):
    """
    Get the books and add to the context
    """
    def __init__(self, obj, context_var):
        self.obj = Variable(obj)
        self.context_var = context_var
 
    def render(self, context):
        content_type = ContentType.objects.get_for_model(
            self.obj.resolve(context))
        # create the template var by adding to context
        context[self.context_var] = \
            Book.objects.filter( # find all books for object
                content_type__pk = content_type.id,
                object_id = self.obj.resolve(context).id
            )
        return ''
 
def books_for_object(parser, token):
    """
    Retrieves a list of books for given object
    {% books_for_object foo_object as book_list %}
    """
    try:
        bits = token.split_contents()
    except ValueError:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    if len(bits) != 4:
        raise TemplateSyntaxError(
            _('tag requires exactly three arguments')
    if bits[2] != 'as':
        raise TemplateSyntaxError(
            _("second argument to tag must be 'as'")
    return BooksForObjectsNode(bits[1], bits[3])
 
def book_form(parser, token):
    """
    Adds a form to the context as given variable
    {% book_form as form %}
    """
    # take steps to ensure template var was formatted properly
    try:
        bits = token.split_contents()
    except ValueError:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    if bits[1] != 'as':
        raise TemplateSyntaxError(
            _("second argument to tag must be 'as'")
    if len(bits) != 3:
        raise TemplateSyntaxError(
            _('tag requires exactly two arguments')
    # get the form
    return BookFormNode(bits[2])
 
class BookFormNode(Node):
    """
    Get the form and add it to the context
    """
    def __init__(self, context_name):
        self.context_name = context_name
    def render(self, context):
        from books.forms import NewBookForm
        form = NewBookForm()
        # create the template var by adding to context
        context[self.context_name] = form
        return ''
 
# register these tags for use in template files
register.tag('books_for_object', books_for_object)
register.tag('book_form', book_form)
register.simple_tag(get_book_form_url)

Add this to your template

To start adding books to an object, add this code to your template and change my_awesome_object_here to the template variable name of your object.

<h2>Books</h2>
{% load book_tags %}
 
{% books_for_object my_awesome_object_here as books %}
{% for book in books %}
 
<a href="{{ book.get_absolute_url }}">{{ book }}</a> -
        {{ book.description }}
 
{% endfor %}
<h2>Add a book</h2>
<form action="{% get_book_form_url my_awesome_object_here %}" method="post">
{% book_form as form %}
{{ form }}
<input type="submit" value="Go" />
</form>

You can get the template tags source code and the code from
the previous post at the
google code project page or by doing

svn co http://django-books.googlecode.com/svn/trunk books

in a directory on the python path.

Related posts:

1. How to Write Reusable Apps for Pinax and Django Pinax is a collection of reusable django apps that…
2. Django RequestContext Example Browsing other peoples’ code is a great way to learn…
3. Quick Thumbnails in Django I normally like to write code myself instead of installing…

The idea came on inauguration day when someone was wondering how the stock market was doing. I had been to http://isobamapresident.com/ earlier in the day and the single serving site idea for the stock market came to mind.

It only took about 3 hours to create after I had the idea.

First 30 minutes: Find free data. Is there a feed or API?
Next 30: I couldn’t find anything useful so I decided to scrape a website using BeautifulSoup.
Next 60: I set up a Django project to put it all together. Created a model to hold the scraped data so I wouldn’t do it every page load. Created a simple view and layed out a template file and simple stylesheet.
Next 15: I found out how to add a method to manage.py so that I could call the scraping from the command line. python manage.py market_parse
Next 15: Debugging and adjusting. I hadn’t looked at the site yet, but there weren’t that many bugs.
Last 30 minutes: add crontab for every 20 minutes on weekdays from 9-5. It took a while to get going because I needed the full python path in the crontab.

So, is the market down?

Pinax is a collection of reusable django apps that brings together features that are common to many websites. It allows developers to focus on what makes their site unique. Here is an example of adding your own functionality to Pinax. It will also be an example of writing a reusable app since every individual app currently in Pinax can be used separately. Also, I’ve bundled the
example files into a google code project.

My example will be to create a list of books and allow them to be tied to any object using
Django’s ContentType framework. The books could be recommended reading for the members of a tribe (pinax group), a class, or anything in your project and will include title, description, and tags (requires
django-tagging). In another post I’ve shown
how to create template tags to make it easy to show the list of books and a form to add a book. Obviously, there is a lot more that could be done with this app, but I will leave it out of the example to keep it simple.

Starting the App

Create a folder in the apps directory or any place that is on the python path (ex. /path/to/pinax/projects/complete_project/apps/books/) and include these files:

• __init__.py even though it might be empty, it is required
• forms.py
• models.py
• urls.py
• views.py

models.py

I will start with creating the model for the project. Below is all of the code I am placing in the file. I’ve added a lot of comments to explain everything that is happening.

#import all of the things we will be using
from django.db                          import models
from tagging.fields                     import TagField
# to help with translation of field names
from django.utils.translation  import ugettext_lazy as _
# to have a generic foreign key for any model
from django.contrib.contenttypes        import generic
# stores model info so this can be applied to any model
from django.contrib.contenttypes.models import ContentType
 
class Book(models.Model):
    """
    The details of a Book
    """
    # fields that describe this book
    name        = models.CharField(_('name'), max_length=48)
    description = models.TextField(_('description'))
 
    # to add to any model
    content_type   = models.ForeignKey(ContentType)
    object_id      = models.PositiveIntegerField()
    content_object = generic.GenericForeignKey('content_type',
        'object_id')
 
    # for the list of tags for this book
    tags        = TagField()
 
    # misc fields
    deleted     = models.BooleanField(default=0)
    created     = models.DateTimeField(auto_now_add=True)
    # so that {{book.get_absolute_url}} outputs the whole url
    @models.permalink
    def get_absolute_url(self):
        return ("book_details", [self.pk])
    # outputs name when printing this object as a string
    def __unicode__(self):
        return self.name

forms.py

Use Django’s ModelForm to create a form for our book model.

from django import forms
from books.models import Book
 
class NewBookForm(forms.ModelForm):
    class Meta:
        model = Book
        exclude = ('deleted', 'content_type',
            'object_id', 'created')

views.py

In this file we create a view to show the details of a book and a view to create a new book for an object.

from django.shortcuts import render_to_response
from django.shortcuts import get_object_or_404
from django.http import HttpResponseRedirect
from django.template import RequestContext
from django.core.urlresolvers import reverse
from django.utils.translation import ugettext_lazy as _
from django.contrib.auth.decorators import login_required
 
from tribes.models import Tribe
from books.models import Book
from django.contrib.contenttypes.models import ContentType
 
@login_required
def new(request, content_type_id, object_id,
            template_name="books/new.html"):
    """
    creates a new book
    """
    from books.forms import NewBookForm
 
    # if a new book was posted
    if request.method == 'POST':
        book_form = NewBookForm(request.POST)
        if book_form.is_valid():
            # create it
            book = book_form.save(commit=False)
            content_type        = \
                ContentType.objects.get(id=content_type_id)
            content_object      = \
                content_type.get_object_for_this_type(
                id=object_id)
            book.content_object = content_object
            book.save()
            request.user.message_set.create(
                message=
                _("Successfully created book '%s'")
                % book.name)
            # send to object page or book page
            try:
                return HttpResponseRedirect(
                    content_object.get_absolute_url()
                )
            except:
                return HttpResponseRedirect(reverse(
                    'book_details', args=(book.id,)))
        # if invalid, it gets displayed below
    else:
        book_form = NewBookForm()
 
    return render_to_response(template_name, {
        'book_form': book_form,
    }, context_instance=RequestContext(request))
 
@login_required
def details(request, book_id,
    template_name="books/details.html"):
    """
    displays details of a book
    """
    book = get_object_or_404(Book, id=book_id)
    return render_to_response(template_name, {
        'book': book,
    }, context_instance=RequestContext(request))

urls.py

To tie our views to some urls, add this to the urls.py file.

from django.conf.urls.defaults import *
from django.conf.urls.defaults import *
 
urlpatterns = patterns('',    
    # new book for object
    url(r'^new/(?P<content_type_id>\d+)/(?P<object_id>\d+)', 
        'books.views.new', name="new_book"),
    # display details of a book
    url(r'^details/(?P<book_id>\d+)$', 'books.views.details', 
        name="book_details"),
)

More Features

The rest of the application is described in the post titled:
How to Write Django Template Tags. You can also check out all of the code from the
google project by doing the following command:

svn co http://django-books.googlecode.com/svn/trunk books

in a directory on the python path.

Related posts:

1. How to Write Django Template Tags Template tags can be useful for making your applications more…
2. How to Add Locations to Python Path for Reusable Django Apps In my previous post I talk about reusable apps, but…
3. How to Display Realtime Traffic Analytics Users of Presskit’n have been asking for traffic statistics on…

It took me a while to figure out how to change from the default, example.com. Maybe it should have been obvious, but I was looking in all the wrong places and every search I did wouldn’t come up with an answer. As I discovered, it isn’t a setting in the settings.py file. It’s something that is in the database. You can change it through the Django admin interface, phpMyAdmin, or how ever you feel comfortable. It’s in the django_site table. When setting SITE_ID in settings.py it is the ID in this table. And this is the information that Site.objects.get_current().domain uses. Which is awesome for rss feeds and e-mails that you send out or anytime you need the domain in a link.

from django.contrib.sites.models import Site
domain = Site.objects.get_current().domain

Related posts:

1. Django Single Sign On or a Solution to Multi-domain Cookies I’ve
2. How to Speed up Your Django Sites with NginX, Memcached, and django-compress A lot of t
3. How to Add Locations to Python Path for Reusable Django Apps In my
   pre

Browsing other peoples’ code is a great way to learn new things about a language or framework. I never made it to the Django docs about Contexts, but the
Pinax developers apparently did and I got a chance to learn this from them. This is a few sections of their code and how they use RequestContext in their apps.

If you are looking at the source of some of their views you might see how they are using it. Here is what it looks like in friends_app.friends

    return render_to_response(template_name, {
        "join_request_form": join_request_form,
        "invites_received": invites_received,
        "invites_sent": invites_sent,
        "joins_sent": joins_sent,
    }, context_instance=RequestContext(request))

So what extactly does context_instance=RequestContext(request) do? I took a look at
the django documentation to find out more. And that led me to check the settings file and I found that there were quite a few things listed in TEMPLATE_CONTEXT_PROCESSORS.

TEMPLATE_CONTEXT_PROCESSORS = (
    "django.core.context_processors.auth",
    "django.core.context_processors.debug",
    "django.core.context_processors.i18n",
    "django.core.context_processors.media",
    "django.core.context_processors.request",
 
    "notification.context_processors.notification",
    "announcements.context_processors.site_wide_announcements",
    "account.context_processors.openid",
    "account.context_processors.account",
    "misc.context_processors.contact_email",
    "misc.context_processors.site_name",
    "messages.context_processors.inbox",
    "friends_app.context_processors.invitations",
    "misc.context_processors.combined_inbox_count",
)

I opened up friends_app.context_processors to see a bit more and it looked like this

from friends.models import FriendshipInvitation
 
def invitations(request):
    if request.user.is_authenticated():
        return {
            'invitations_count':
            FriendshipInvitation.objects.filter(
            to_user=request.user, status="2").count(),
        }
    else:
        return {}

This means that every view that has context_instance=RequestContext(request) in it will call the above function since it is listed in settings.py and it will provide the template variable, {{ invitations_count }}.

Using RequestContext makes it easy to have the common template variables available on every page and I will have to start using it more in my apps. So make sure you have from django.shortcuts import render_to_response and from django.template import RequestContext in your file and add your context processor to the settings file and you should be ready to add template vars to your contexts.

Related posts:

1. How to Write Reusable Apps for Pinax and Django
   Pinax is
2. How to Write Django Template Tags Template t
3. Quick Thumbnails in Django I normally

I’ve been curious to learn more about screen scraping for some time. And then I heard about a python script that is great for parsing html. Since I’ve also been learning python, I thought now was the perfect time to explore some scraping.

In the past I had some trouble with using php to parse the magic the gathering official site for new card info when working on my mtg card database. I didn’t spend much time trying to figure that out, but using python I didn’t have a problem.

After copying Beautiful Soup to my python path I started typing in some python at the command line.

from BeautifulSoup import BeautifulSoup as BSoup
import urllib
url  = 'http://ww2.wizards.com/gatherer/Index.aspx?setfilter=Shards%20of%20Alara&output=Spoiler'
html = urllib.urlopen(url).read()
soup = BSoup(html)
for tr in soup.fetch('tr'):
    if tr.td:
        print tr.td.string

This would output all of the magic card names on the page (and some other stuff). Here is another example: getting image urls when knowing the value of the id attribute on the img tags.

url  = 'http://ww2.wizards.com/gatherer/CardDetails.aspx?&id=175000'
html = urllib.urlopen(url).read()
soup = BSoup(html)
for img in soup.findAll(id='_imgCardImage'):
    print img['src']

With a little more time I could get all the cards and their images and fill up my database. I just have to find the time now.

Update 12/28/8

I just heard about Scrapy. Now I need to try it out with a project.

Both Debian and Ubuntu make it really simple to get a server up and running. I was trying a few different Machine Images on Amazon and I found myself repeating a lot of things so I wanted to put them here for reference for those who might find it useful.

With a fresh image, the first thing to do is update apt-get.

apt-get update && apt-get upgrade -y

Then grab all of the software to use.

apt-get install -y xfsprogs mysql-server  apache2  libapache2-mod-python  python-mysqldb  python-imaging  python-django  subversion php5  phpmyadmin

xfsprogs is for formatting an Elastic Block Store volume and may not be needed in all cases.

I like to check out the latest version of Django from their repository, it makes it easier to update it later. This also starts a project named myproject (this name is used later).

cd /usr/lib/python2.5/site-packages
svn co http://code.djangoproject.com/svn/django/trunk/django django
ln -s /usr/lib/python2.5/site-packages/django/bin/django-admin.py /usr/local/bin
cd /var/www
django-admin.py startproject myproject

Now to edit the apache config to tell it about our project.

cd /etc/apache2
nano httpd.conf

Add the following to set up python to run the django files and php to run the phpmyadmin files. There is also an example of serving static files. Change where it says myproject if you used a different name.

<Location "/">
    SetHandler python-program
    PythonHandler django.core.handlers.modpython
    SetEnv DJANGO_SETTINGS_MODULE myproject.settings
    PythonOption django.root /myproject
    PythonDebug On
    PythonPath "['/var/www'] + sys.path"
</Location>
 
 
Alias /adm_media/ /usr/lib/python2.5/site-packages/django/contrib/admin/media/
<Location "/adm_media/">
    SetHandler None
</Location>
 
Alias /files/ /var/www/myproject/files/
<Location "/files/">
    SetHandler None
</Location>
 
Alias /phpmyadmin/ /usr/share/phpmyadmin/
<Location "/phpmyadmin/">
    SetHandler None
</Location>

Restart apache for it to use the new configuration.

/etc/init.d/apache2 restart

The only thing left to do is set up the database. If Ubuntu had you set up a root password already, add -p to the end of the following command to use it.

mysql

There are some users in mysql without username, it is best to remove those.

drop user ''@'localhost';

Do that for each host that has a blank username. Use the following to see all users.

SELECT user, host FROM mysql.user;

Create a database and add a user.

CREATE DATABASE db_name;
GRANT ALL ON db_name.* to user_name WITH GRANT OPTION;
SET PASSWORD FOR user_name = password('psswdhere');

If root doesn’t have a password yet, use the above commant with root as the username.

Amazon has a page about
how to use EBS with MySQL, but
there are reported issues with using Debian Lenny and EBS.

I ran into some trouble with python2.4 and the django code I was using. The previous server had 2.5 and I didn’t notice any problems, so I tried upgrading to 2.5 and changing which version of python Debian uses as default (this was on Debian Etch). I was having some difficulty getting a few of the site-packages to work with 2.5 by default (like mod_python), so I decided to move to Debian Lenny even though it isn’t as supported. While doing that I ran into a problem where it doesn’t work well with xfs and Amazon’s Elastic Block Store. They are looking into the matter, but while trying to figure that out, I realized that AWS doesn’t come with support. There is an extra package you have to purchase which starts at $100 a month.

That made Amazon look less awesome since I know I am going to need some support at some point. I decided to compare prices and features around again. I ended up revisiting Slicehost since I knew a lot more about setting up a server than I did before.

I posted the steps that I took to set up apache, mysql, django, and a few other things on a clean ubuntu machine on Code Spatter.

Now I have a WebFaction account for testing and subversion hosting and I’m using the Slicehost account for the live version of the site.

Subversion makes it easy to commit on one server and update on the other once it is stable. I should explore a distributed version control system like git since it might help out with this in the future.

Update October 21, 2008

The AWS developer community seems to be a good alternative to having direct support from amazon. The people there are knowledgeable and amazon reps post frequently. Here is a quote from someone at amazon about the issue I was having

We are still investigating the issue and will post an analysis a little later and a workaround.  Basically the problem revolves around the interaction between very specific kernel versions, XFS and our version of Xen.

Even though my slice is running fine, I will still be keeping AWS in mind.

Update May 7, 2009

Some people have posted some solutions on the developer community. I haven’t tested them, but I will look into it if I need to use Amazon again.