In this week I learned a lot of things about information retrieval, text categorization, natural language processing and machine learning. But the most relevant thing is: the principles are easy, their implementation is not. The fact is that most of the techniques are relatively simple but you usually have to deal with very large datasets and this could be challenging, since one of the main requirements about information retrieval is time. It’s really much more important that you give less results in one second rather than giving better results in one hour. No one will ever care to use your system if it takes an hour to get some result. And if you’re considering to store your data in a database forget about normalization, it wouldn’t really take you anywhere.

Talking about storing informations, you know that if you’re dealing with documents most of the words are the so called stop words. Those stop words are words that doesn’t really mean anything, but they help the readers to get a better text flux. Classic examples of stop words are articles like “the”, “a”, “an” or logic connectors like “or” and “and”. These words are so common that their presence is quite useless since they’re are… everywhere. If you’re going to study information retrieval than you’ll learn about a weighting technique called tf-idf that gives a weight near to 0 to these words, but since you’d probably use a reverse index for words (an index that given a word, tells you in which documents that word appears) you can understand that this would take a lot of space if you’re going to include stop words.

So one of the biggest issues until now is that you’re going to deal with extremely large datasets, so you have to strip as many things as possible. Now consider those words: “fishing”, “fishes”, “fish”. They all talk about “fish”, and an user that is searching for “fish” would probably be interested in “fishes” or “fishing” as well. Additionally, it’s useless to store three words that are almost identical. So here comes the stemming that, by quoting the related wikipedia page, is the process for reducing inflected (or sometimes derived) words to their stem, base or root form – generally a written word form. Fortunately, if you’re dealing with english texts, there’s the Porter algorithm that is the state-of-the-art algorithm for this sort of things. But that works only with english, so if your documents are written in another language or they are written in multiple languages, things are going to be complicated.

This leads to think about the problem of the language identification. How do you know if some text is written in a language or in another just by looking at it? Of course you can describe the document’s language with some kind of meta tagging, but not all the documents have this kind of description, just think about the web. There are some kind of statistical methods based upon the classification of n-grams but I haven’t deeply investigated about them yet, so I can’t really say anything.

Now you got your collection of documents that match a certain query. Now: how do you know what document is more relevant than another (in other words: how do you rank pages)? You got two alternatives (well, probably more, but I know just these at this moment): the tf-idf that we said above and the cosine similarity. The latter is an interesting one: consider the tf-idf vectors of the documents, then consider the query as a document too. Now plot those tf-idf vectors and measure their cosine of the angle between them. The more you’re near to 1, the more relevant is the document.

There are a lot of other important things that need to be said like the precision and recall concept, but that’s enough for now. I’ll talk about this another time.

Anyway I’m doing an experimental project named django searchable. It’s a pluggable app for django that implements an information retrieval engine based on tf-idf weighting. Play with it if you’re brave enough.

It has a very basic start/stop/restart interface like normal startup scripts. Let see an example of basic usage:

kratorius@becks:~/prj$ sudo sh startserver.sh start
Starting fcgi process... done!
kratorius@becks:~/prj$ sudo sh startserver.sh stop
kratorius@becks:~/prj$ sudo sh startserver.sh restart
fcgi process is not running
kratorius@becks:~/prj$ sudo sh startserver.sh start
Starting fcgi process... done!
kratorius@becks:~/prj$ sudo sh startserver.sh restart
Starting fcgi process... done!

And here it is:

#!/bin/bash
# Start and stop a django fcgi process
# Giuliani Vito Ivan <giuliani.v@gmail.com>

# project directory
PROJDIR=`pwd`

# user owner (usually www-data)
USER_OWNER="www-data"

# group owner (usually www-data)
GROUP_OWNER="www-data"

# extra python path (leave empty if unneeded)
PYTHONPATH="../python:.."

# do not edit anything below
PIDFILE="$PROJDIR/technosec.pid"
SOCKET="$PROJDIR/technosec.sock"

start_fcgi()
{
	if [ -f $PIDFILE ] || [ -f $SOCKET ]; then
		echo "The fcgi process is already running, please stop"
		echo "that before running another process"
	else
		echo -n "Starting fcgi process... "

		nohup /usr/bin/env - \
			PYTHONPATH=$PYTHONPATH \
			python manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE > /dev/null 2>&1 &
		if [ $? -eq 0 ]; then
			sleep 1
			chown $USER_OWNER:$GROUP_OWNER $SOCKET
			chown $USER_OWNER:$GROUP_OWNER $PIDFILE
			echo "done!"
		else
			echo "failed!"
		fi

		return $?
	fi

	return 1
}

stop_fcgi()
{
	cd $PROJDIR
	if [ -f $PIDFILE ]; then
		kill `cat -- $PIDFILE`
		rm -f -- $PIDFILE
		rm -f -- $SOCKET

		return 0
	else
		echo "fcgi process is not running"
		return 1
	fi
}

restart_fcgi()
{
	stop_fcgi
	if [ $? -eq 0 ]; then
		start_fcgi
	fi
}

case "$1" in
	start)
		start_fcgi
		;;
	stop)
		stop_fcgi
		;;
	restart)
		restart_fcgi
		;;
	*)
		echo "Usage: $0 {start|stop|restart}"
		exit 1
		;;
esac

exit 0

Just in case you may want to download it, here it is.

I did this for generating a django quick reference (here it is) since it has a lot of functions and I know what’s their purpose, but I can never remember the names (and now two A4 papers are right in front of me).

If you’re interested in this, check out the google code project page and grab your copy from the SVN repository.

There are tons of things that can be changed/optimized (i.e.: add some optional short explanation of the function, add more examples, easier way to change colors, …) but now the code is working quite well so that can be already useful to the people out there.

The competition began very well, I worked normally for the first part of the day but then I had to stop for a while. When I came back, svn and djangodash website was not working anymore. I initially thought that it was some connection issue but when I saw that other sites were working properly so they definitely had some problems.

I just waited, then gone sleeping. In the morning I received in my mailbox a message that informed me of a big power outage in The Planet datacenter where webfaction hosts a lot of their server (among the which there was the djangodash one) caused by power generator’s explosion. Then the competition has been delayed for other two days, so I decided to take a breath and wait ‘till the svn would came back. But that didn’t happen on Sunday, so after a while I chosen (as the mail suggested) to work locally without committing anything at least until the svn return.

Then Monday came and I had other things to do, so I had to postpone djangodash for the evening when I’d freed myself from other, most urgent things. On Monday I did a very little coding, as well on Tuesday. So at the end of competition I cannot complete my project, and not even reach the 50% milestone.

Today I discovered that I was one of the winners (ok not really, 6th place was not really a good place, but at least I tried) and I really have to thanks the organizers for this event and hope to join another djangodash next year. Maybe, as I said to one of them in an email thread, hosting the site/svn in two different datacenters, just to be insured against eventual thunderstorms, tornado, earthquakes and so on…). I have to say that I really enjoyed the whole thing, and hope to have more competitors next year!

If you want to get more news about final process of djangodash with some stats, read this article on the Toast Driven website (that’s the company that ran the dash).

[…] is a chance for Django enthusiasts to flex their coding skills a little and put a fine point on “perfectionists with deadlines” by giving you a REAL deadline. 48 hours from start to stop to produce the best app you can and have a little fun in the process.

I’ll be participating, so if you haven’t registered yet, do it now! And don’t forget to check out the cool prizes :)

The main issue is that even if an ImageField has a width_field/height_field option that refers to (presumably) integer fields that will be auto-filled with the image size, we can’t ran a validator across those fields (we can do so only in a form, but my problem was to validate the image in the admin area). So we have to manually load the image in memory and run a custom validator that uses PIL to get the needed information and validate the image.

Here’s the resulting code for the model:

class FImage(models.Model):
    ...
    img = models.ImageField(upload_to='images', blank=False,
            verbose_name=_("Image"),
            validator_list=[custom_validators.ImageLarge(620, 250,
            _("The image must be 620x250px large!")), ],
            help_text=_("Image must be 620x250px large."))
    ...

And for the validator (that you should put in some separated file):

from django.utils.translation import gettext_lazy as _
from django.core import validators
import StringIO
from PIL import Image

class ImageLarge(object):
    def __init__(self, width, height, error_message=_("The image isn't as large as expected!")):
        self.w, self.h, self.error_message = width, height, error_message

    def __call__(self, field_data, all_data):
        im = Image.open(StringIO.StringIO(field_data['content']))
        if im.size[0] != self.w or im.size[1] != self.h:
            raise validators.ValidationError, self.error_message 

The main issue with this solution is that we store the entire image content in memory for processing through PIL. Additionally I don’t know if it will still work when the #2070 ticket’s patch will be included in trunk (and I didn’t checked with the
actual existing patch).

If you want, you can read the full discussion on django-users.

• me (to be precise, my company) stopped development of PHP sites about one year ago in favor of Python
• we release the web site’s code to them
• for this project, we haven’t been asked any kind of future support; this means that when the site is finished, we won’t touch the product anymore (unless they don’t pay us to do the modifies they need)
• but they don’t want to pay us to these modifies, because they have their internal IT technician
• their technician knows only PHP (and he never even known the Python’s existence until yesterday)

So I had to illustrate why me and my company chosen Python for our web development needs, and here’s a summary of what I told them yesterday. Note that I’m not talking about why a language is better than the other, because this would move us in another direction, but why we chosen Python as our main programming language (even if this unconsciously lead us to say why, for us, Python is better than PHP, but that’s another story) :

• Time: remember that time is money. If I’d build an application in PHP, I’d spend about 1/3 of the time more if I’d develop the same application in Python
• Frameworks: nowadays all the popular languages have their web frameworks; Ruby has Ruby on Rails, PHP has Symfony, php.MVC, Python has Django, CherryPy, Pylons, and so on. But none of them (apart from rails, but we’re talking about PHP Vs Python) goes near to the completeness and functionality of django. And this take us at the previous point: less time in develop the same application
• Future modifies: even if for this project this isn’t the case, it’s much easier to modify a Python application than a PHP one due to the syntax of the language and its strong OOP orientation. You may argue that PHP 5 introduced a deep OOP support too, but that’s not the same thing and you know. PHP is born as procedural programming language, and even if we had OOP introduction in version 5, it doesn’t even comes near to Python under this point of view

With this I’m not saying that PHP is useless: what I mean is that Python is more convenient under the (our) business point of view. So if someone ask me why I use Python to do the same thing I can make with PHP, the answer will be: because with Python I can make the same thing in a shorter time and, consequently, with a lower budget.

If at first glance this couldn’t look so useful to you, well, this hasn’t been true for me since it was exactly that kind of stuff I was looking for. What I was trying to do was to implement some kind of auto-installer for an application that I’m writing, and in order to do so I had to run the set up after that the syncdb command is issued. So the main problem was: how do I know when a user does the syncdb in a non-intrusive way?

After some googling, I found that in django exists the semi-hidden feature of signals, so I began exploring them. The only thing you can look at on the official site is a page on their wiki, and additionally there are some cool articles over the net. Anyway, if you want to know a fast way to catch the syncdb command just follow up the reading.

First of all, let say that django looks for a file named management.py in every installed application’s directory whenever the syncdb command is ran. In this file we’ll write our management functions and we’ll link those functions to django through its signal system.

To do so we need to import some basic django’s modules:

from django.dispatch import dispatcher
from django.db.models import signals

With these modules we have everything to handle the django’s signals, and so here we go:

dispatcher.connect(do_something_with_the_signal, signal=signals.post_syncdb, sender=model)

With that function we connect our function do_something_with_the_signal to the django’s signal handler. A word apart has to be given to the sender attribute. In that way we say that our function must be called only when the appropriate model gets called in the syncdb phase.

The signal handling’s function must be defined as following:

def do_something_with_the_signal(app, created_models, verbosity, **kwargs):

So let describe the parameter list:

• app: the dispatcher sender’s attribute
• created_models: a list of models that have been created with this syncdb
• verbosity: django’s verbosity level
• kwargs: a variable-length’s list of optional arguments

Let suppose we want to create a function that inserts the settings.LANGUAGES list into a model (as I had to do for a little work). Here is how the code looks like:

from django.conf import settings
from django.dispatch import dispatcher
from django.db.models import signals
from project.translatable import models as transmodel

def create_language_list(app, created_models, verbosity, **kwargs):
    for lang in settings.LANGUAGES:
        lcode, literal = lang

        if transmodel.Language in created_models:
            print "Adding %s (%s)..." % (literal, lcode)
            new_lang = transmodel.Language(code=lcode, name=literal)
            new_lang.save()

dispatcher.connect(create_language_list, signal=signals.post_syncdb, sender=transmodel)

It’s worth noting here that using the print statement is perfectly OK (this is done by django’s auth system for example). Another thing to note is that we check that our model has been created in that syncdb’s phase. I do so because in this way the model is filled just the first time the model is created.

Another use of this stuff is to insert initial raw data into the model by using django’s ORM and not by using raw SQL. In this way we’ll be database-backend indipendent and we’ll have a transparent way to have initial data in our models.