I am the programmer behind IvyLees The network for PR and business
and Presskit'n Do-It-Yourself Public Relations.
I developed the sites with django and also administrate the servers.
I am one of the developers in Match Strike: we build websites.
There are two more django apps that could help reduce http requests. I didn’t find these until after posting this. http://code.google.com/p/django-sprites/ and http://github.com/rnk/django-media-bundler/tree… will help create image sprites so that your css background images are all one file request. CSS sprites are explained in more detail here http://alistapart.com/articles/sprites
A lot of these steps will speed up any kind of application, not just django projects, but there are a few django specific things. Everything has been tested on
IvyLees which is running in a Debian/Ubuntu environment.
These three simple steps will speed up your server and allow it to handle more traffic.
Yahoo has developed a
firefox extension called
YSlow. It analyzes all of the traffic from a website and gives a score on a few categories where improvements can be made.
It recommends reducing all of your css files into one file and all of your js files into one file or as few as possible. There is a pluggable, open source django application available to help with that task. After setting up
django-compress, a website will have css and js files that are minified (excess white space and characters are removed to reduce file size). The application will also give the files version numbers so that they can be cached by the web browser and won’t need to be downloaded again until a change is made and a new version of the file is created.
How to setup the server to set a far future expiration is shown below in the lightweight server section.
Django makes it really simple to set up caching backends and memcached is easy to install.
sudo aptitude install memcached, python-setuptools
We will need setuptools so that we can do the following command.
sudo easy_install python-memcached
Once that is done you can start the memcached server by doing the following:
sudo memcached -d -u www-data -p 11211 -m 64
-d will start it in daemon mode, -u is the user for it to run as, -p is the port, and -m is the maximum number of megabytes of memory to use.
Now open up the settings.py file for your project and add the following line:
CACHE_BACKEND = 'memcached://127.0.0.1:11211/'
Find the MIDDLEWARE_CLASSES section and add this to the beginning of the list:
'django.middleware.cache.UpdateCacheMiddleware',
and this to the end of the list:
'django.middleware.cache.FetchFromCacheMiddleware',
For more about caching with django see the
django docs on caching. You can reload the server now to try it out.
sudo /etc/init.d/apache2 reload
To make sure that memcached is set up correctly you can telnet into it and get some statistics.
telnet localhost 11211
Once you are in type stats and it will show some information (press ctrl ] and then ctrl d to exit). If there are too many zeroes, it either isn’t working or you haven’t visited your site since the caching was set up. See
the memcached site for more information.
Apache has some overhead involved that makes it good for serving php, python, or ruby applications, but you do not need that for static files like your images, style sheets, and javascript. There are a few options for lightweight servers that you can put in front of apache to handle the static files.
Lighttpd (lighty) and
nginx (engine x) are two good options. Adding this layer in front of your application will act as an application firewall so there is a security bonus to the speed bonus.
There is this guide to
install a django setup with nginx and apache from scratch. If you followed
my guide to set up your server or already have apache set up for your application, then there are a few steps to get nginx handling your static files.
sudo aptitude install nginx
Edit the config file for your site (sudo nano /etc/apache2/sites-available/default) and change the port from 80 to 8080 and change the ip address (might be *) to 127.0.0.1. The lines will look like the following
NameVirtualHost 127.0.0.1:8080 <VirtualHost 127.0.0.1:8080>
Also edit the ports.conf file (sudo nano /etc/apache2/ports.conf) so that it will listen on 8080.
Listen 8080
Don’t restart the server yet, you want to configure nginx first. Edit the default nginx config file (sudo nano /etc/nginx/sites-available/default) and find where it says
location / {
root /var/www/nginx-default;
index index.html index.htm;
}
and replace it with
location / {
proxy_pass http://192.168.0.180:8080;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;
}
location /files/ {
root /var/www/myproject/;
expires max;
}
/files/ is where I’ve stored all of my static files and /var/www/myproject/ is where my project lives and it contains the files directory.
Set static files to expire far in the future
expires max; will tell your users’ browsers to cache the files from that directory for a long time. Only use that if you are use those files won’t change. You can use expires 24h; if you aren’t sure.
Configure gzip
Edit the nginx configuration to use gzip on all of your static files (sudo nano /etc/nginx/nginx.conf). Where it says gzip on; make sure it looks like the following:
gzip on;
gzip_comp_level 2;
gzip_proxied any;
gzip_types text/plain text/html text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript;
The servers should be ready to be restarted.
sudo /etc/init.d/apache2 reload sudo /etc/init.d/nginx reload
If you are having any problems I suggest reading through
this guide and seeing if you have something set up differently.
Those three steps should speed up your server and allow for more simultaneous visitors. There is a lot more that can be done, but getting these three easy things out of the way first is a good start.
Related posts:
I don’t think there is anyway to do it in the database, but you can still put the logic in your application. If you are using django, there are signals that will help get that done. http://docs.djangoproject.com/en/1.0/topics/sig… If you aren’t then you will need to program it into your project in some other way.
That is a good post. I’ve been wanting to use nginx since you mentioned it 5 months ago and now I have no excuse for not using it.
In my
previous post I talk about reusable apps, but I don’t really explain it that much. If you have an app that might be useful in another project, it’s best to not refer to the project name in the application so you don’t have to search and remove it when adding to another project. To never refer to your project name in your app’s code, you will need to put your app on the python path. I usually do project_folder/apps/app_folder so apps will need to be a location that python is checking when you are importing so that importing looks like the following:
from appname.filename import foo
There are a few places you might need to add an apps folder to the pythonpath.
This will add the apps directory in your project directory to the beginning of the path list. This will allow manage.py syncdb and manage.py runserver to know that the apps folder should be added.
import os import sys PROJECT_ROOT = os.path.dirname(__file__) sys.path.insert(0, os.path.join(PROJECT_ROOT, "apps"))
That should be all you need to do to get most django projects working with your reusable apps, but if for any reason you need add to the path with mod python or mod wsgi, the following should work.
In the
setting-up-everything post I show an example httpd.conf file. In your apache configuration you will probably see something similar to what is below. To add the location /var/www/myproject/apps to the PythonPath I added it in the list.
SetHandler python-program PythonHandler django.core.handlers.modpython SetEnv DJANGO_SETTINGS_MODULE myproject.settings PythonOption django.root /myproject PythonDebug On PythonPath "['/var/www','/var/www/myproject/apps'] + sys.path"
If you use mod wsgi instead of mod python, your apache config will be loading a wsgi file with a line like this WSGIScriptAlias /var/www/myproject/myproject.wsgi. You will need to edit that file to add to the path (django’s site has an
example file).
sys.path.append('/var/www/myproject/apps')
You never know when you might want to use an app in another project, so always try to keep from mentioning the project name anywhere in the applications.
Related posts:
I found
one library that was linked everywhere, but it wasn’t working for me. I was always getting 400 Bad Request when using it. Chris Conover was able to get the following code working.
import urllib2 protocol = 'https://' url = 'example.com' command = '/projects.xml' headers = {'Accept' : 'application/xml', 'Content-type' : 'applications/xml'} username = 'x' password = 'y' # Setup password stuff passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, url, username, password) authhandler = urllib2.HTTPBasicAuthHandler(passman) opener = urllib2.build_opener(authhandler) urllib2.install_opener(opener) # Send the request and get response req = urllib2.Request(protocol + url + command, None, headers) response = urllib2.urlopen(req) results = response.read() print results
I thought it was a problem with how the authorization was formed so based on the above code I modified the old basecamp.py file and I was able to get a response. The following is what I changed.
Around line 64
def __init__(self, username, password, protocol, url): self.baseURL = protocol+url if self.baseURL[-1] == '/': self.baseURL = self.baseURL[:-1] passman = urllib2.HTTPPasswordMgrWithDefaultRealm() passman.add_password(None, url, username, password) authhandler = urllib2.HTTPBasicAuthHandler(passman) self.opener = urllib2.build_opener(authhandler)
And around line 142
path = '/projects.xml'
With that I was able to use basecamp.py to retrieve a list of projects. Other modifications may be needed for other features, but that was all I planned on using.
Here is an example of using
ElementTree to parse the XML response to get the names of all of the projects returned from basecamp.
import elementtree.ElementTree as ET from basecamp import Basecamp protocol = 'https://' url = 'example.com' username = 'x' password = 'y' bc = Basecamp(username, password, protocol, url) projects = bc.projects() tree = ET.fromstring(projects) tags = tree.getiterator(tag='project') for t in tags: project_name = t.findtext('name')
Related posts:
I didn’t walk out on any and I probably should have a few times. I’ll have a backup plan next time.
I didn’t hear much about
SXSW before going. I knew there would be a lot of presentations to see and there would be a few parties. I never expected the parties to be more valuable than the presentations and I was completely surprised by that.
The evening events were a great chance to meet people that are doing awesome things. There was the
unclasses guy and the
awe.sm guy that I met through
Tim. And then there was someone doing something more awesome than I ever could. He is a
professional contest winner. He enters funny videos into contests and wins things like trips around the world where he will make more videos and win more contests.
I met a lot of the Orlando crowd while in another city. I met and followed
Eric a few months ago and through him I was able to meet some of the people that go to
Florida Creatives happy hour. So that was a great chance to get to know everyone better.
There was also a presentation from the
rails envy podcast guy, Gregg Pollack from Orlando. Gregg’s was the only one I went to that wasn’t a panel. The panels could be a bit boring sometimes, but they gave a chance to hear from about 5 people with experience on the topic. The higher education panel started bad just like some of the other panels, but it got better as it went since they got into the important stuff. It would seem that I am still passionate about improving education and I hope that I will be able to work on something at UCF soon that will do just that.
There were a few other panels that weren’t that bad like How to Rawk SXSW where the panelists were experienced SXSWers and gave advice to the noobs. After my visit, the advice I would give is “Be ready to network” because that is the most important part of the conference. And the last panel I attended was how to rawk the rest of the year which had some advice like keep in contact with the people that you’ve met here. One easy way of doing that is finding out who they are on twitter (everyone at the event was on twitter by the end). One other piece of advice from the panel was the parties aren’t the only place to meet people, hallways and streets work well too. Maybe that would have been better advice at the beginning, but it arose from a question.
Hopefully I will get to run into a lot of the Orlando people again at the
happy hours and of course I’ll be at
BarCampOrlando on April 18th.
Related posts:
Template tags can be useful for making your applications more reusable by other projects. For this example I will be adding to the
books project that I started in a previous post. Also, I’ve bundled the
example files into a google code project.
Start off by creating a folder called templatetags in your app directory and create two files in it. The first one named __init__.py and the second book_tags.py. There’s 3 things that we need to accomplish with our template tags. The first is to create a tag that will output the url for the action of the form. For example, {% get_book_form_url foo_object %}Next we need to get the form and assign it to a template variable that can be specified by the template variable. For example, {% book_form as bar_var %}. And the third template tag will get the books for an object and place in a template variable. For example, {% books_for_object foo_object as bar_var %}.
from django.template import Library, Node, TemplateSyntaxError from django.template import Variable, resolve_variable from django.utils.translation import ugettext as _ from django.contrib.contenttypes.models import ContentType from django.core.urlresolvers import reverse from books.models import Book register = Library() def get_contenttype_kwargs(content_object): """ Gets the basic kwargs necessary for form submission url """ kwargs = {'content_type_id': ContentType.objects.get_for_model(content_object).id, 'object_id': getattr(content_object, 'pk', getattr(content_object, 'id')), } return kwargs def get_book_form_url(content_object): """ prints url for form action """ kwargs = get_contenttype_kwargs(content_object) return reverse('new_book', kwargs=kwargs) class BooksForObjectsNode(Node): """ Get the books and add to the context """ def __init__(self, obj, context_var): self.obj = Variable(obj) self.context_var = context_var def render(self, context): content_type = ContentType.objects.get_for_model( self.obj.resolve(context)) # create the template var by adding to context context[self.context_var] = \ Book.objects.filter( # find all books for object content_type__pk = content_type.id, object_id = self.obj.resolve(context).id ) return '' def books_for_object(parser, token): """ Retrieves a list of books for given object {% books_for_object foo_object as book_list %} """ try: bits = token.split_contents() except ValueError: raise TemplateSyntaxError( _('tag requires exactly two arguments') if len(bits) != 4: raise TemplateSyntaxError( _('tag requires exactly three arguments') if bits[2] != 'as': raise TemplateSyntaxError( _("second argument to tag must be 'as'") return BooksForObjectsNode(bits[1], bits[3]) def book_form(parser, token): """ Adds a form to the context as given variable {% book_form as form %} """ # take steps to ensure template var was formatted properly try: bits = token.split_contents() except ValueError: raise TemplateSyntaxError( _('tag requires exactly two arguments') if bits[1] != 'as': raise TemplateSyntaxError( _("second argument to tag must be 'as'") if len(bits) != 3: raise TemplateSyntaxError( _('tag requires exactly two arguments') # get the form return BookFormNode(bits[2]) class BookFormNode(Node): """ Get the form and add it to the context """ def __init__(self, context_name): self.context_name = context_name def render(self, context): from books.forms import NewBookForm form = NewBookForm() # create the template var by adding to context context[self.context_name] = form return '' # register these tags for use in template files register.tag('books_for_object', books_for_object) register.tag('book_form', book_form) register.simple_tag(get_book_form_url)
To start adding books to an object, add this code to your template and change my_awesome_object_here to the template variable name of your object.
<h2>Books</h2>
{% load book_tags %}
{% books_for_object my_awesome_object_here as books %}
{% for book in books %}
<a href="{{ book.get_absolute_url }}">{{ book }}</a> -
{{ book.description }}
{% endfor %}
<h2>Add a book</h2>
<form action="{% get_book_form_url my_awesome_object_here %}" method="post">
{% book_form as form %}
{{ form }}
<input type="submit" value="Go" />
</form>
You can get the template tags source code and the code from
the previous post at the
google code project page or by doing
svn co http://django-books.googlecode.com/svn/trunk books
in a directory on the python path.
Related posts:
Pinax is a collection of reusable django apps that brings together features that are common to many websites. It allows developers to focus on what makes their site unique. Here is an example of adding your own functionality to Pinax. It will also be an example of writing a reusable app since every individual app currently in Pinax can be used separately. Also, I’ve bundled the
example files into a google code project.
My example will be to create a list of books and allow them to be tied to any object using
Django’s ContentType framework. The books could be recommended reading for the members of a tribe (pinax group), a class, or anything in your project and will include title, description, and tags (requires
django-tagging). In another post I’ve shown
how to create template tags to make it easy to show the list of books and a form to add a book. Obviously, there is a lot more that could be done with this app, but I will leave it out of the example to keep it simple.
Create a folder in the apps directory or any place that is on the python path (ex. /path/to/pinax/projects/complete_project/apps/books/) and include these files:
I will start with creating the model for the project. Below is all of the code I am placing in the file. I’ve added a lot of comments to explain everything that is happening.
#import all of the things we will be using from django.db import models from tagging.fields import TagField # to help with translation of field names from django.utils.translation import ugettext_lazy as _ # to have a generic foreign key for any model from django.contrib.contenttypes import generic # stores model info so this can be applied to any model from django.contrib.contenttypes.models import ContentType class Book(models.Model): """ The details of a Book """ # fields that describe this book name = models.CharField(_('name'), max_length=48) description = models.TextField(_('description')) # to add to any model content_type = models.ForeignKey(ContentType) object_id = models.PositiveIntegerField() content_object = generic.GenericForeignKey('content_type', 'object_id') # for the list of tags for this book tags = TagField() # misc fields deleted = models.BooleanField(default=0) created = models.DateTimeField(auto_now_add=True) # so that {{book.get_absolute_url}} outputs the whole url @models.permalink def get_absolute_url(self): return ("book_details", [self.pk]) # outputs name when printing this object as a string def __unicode__(self): return self.name
Use Django’s ModelForm to create a form for our book model.
from django import forms from books.models import Book class NewBookForm(forms.ModelForm): class Meta: model = Book exclude = ('deleted', 'content_type', 'object_id', 'created')
In this file we create a view to show the details of a book and a view to create a new book for an object.
from django.shortcuts import render_to_response from django.shortcuts import get_object_or_404 from django.http import HttpResponseRedirect from django.template import RequestContext from django.core.urlresolvers import reverse from django.utils.translation import ugettext_lazy as _ from django.contrib.auth.decorators import login_required from tribes.models import Tribe from books.models import Book from django.contrib.contenttypes.models import ContentType @login_required def new(request, content_type_id, object_id, template_name="books/new.html"): """ creates a new book """ from books.forms import NewBookForm # if a new book was posted if request.method == 'POST': book_form = NewBookForm(request.POST) if book_form.is_valid(): # create it book = book_form.save(commit=False) content_type = \ ContentType.objects.get(id=content_type_id) content_object = \ content_type.get_object_for_this_type( id=object_id) book.content_object = content_object book.save() request.user.message_set.create( message= _("Successfully created book '%s'") % book.name) # send to object page or book page try: return HttpResponseRedirect( content_object.get_absolute_url() ) except: return HttpResponseRedirect(reverse( 'book_details', args=(book.id,))) # if invalid, it gets displayed below else: book_form = NewBookForm() return render_to_response(template_name, { 'book_form': book_form, }, context_instance=RequestContext(request)) @login_required def details(request, book_id, template_name="books/details.html"): """ displays details of a book """ book = get_object_or_404(Book, id=book_id) return render_to_response(template_name, { 'book': book, }, context_instance=RequestContext(request))
To tie our views to some urls, add this to the urls.py file.
from django.conf.urls.defaults import * from django.conf.urls.defaults import * urlpatterns = patterns('', # new book for object url(r'^new/(?P<content_type_id>\d+)/(?P<object_id>\d+)', 'books.views.new', name="new_book"), # display details of a book url(r'^details/(?P<book_id>\d+)$', 'books.views.details', name="book_details"), )
The rest of the application is described in the post titled:
How to Write Django Template Tags. You can also check out all of the code from the
google project by doing the following command:
svn co http://django-books.googlecode.com/svn/trunk books
in a directory on the python path.
Related posts:
