Recently I needed to show a heat map of a quite a lot of coordinate points for a little project of mine that ended up in a data visualization contest (that unfortunately I didn’t win, even though I made to the finalists). The idea was to show the distribution of the georeferenced wikipedia pages through a heat map, so when I first heard about openheatmap.com I knew it was the tool to use. OpenHeatMap.com is an excellent project by Pete Warden that takes a dataset as a CSV, Excel or Google Spreadsheet file and convert it to a nice, browsable heat map presentation.

The first step was to obtain I dataset I could work on. I first tried to work directly onto the whole wikipedia database dump, extracting all the georeferenced pages in a smaller dataset. I actually succeeded but this work included only the english georeferenced pages. Also, extracting and converting coordinates to a common format would have been a real pain. So instead I decided to use the dump from the wikipedia-world project that already included data in a CSV file from all the downloadable wikipedia dumps which include other languages other than english. This dataset include roughly 1.300.000 points, so I had to narrow down some options to process it.

Once I had the dataset ready and knew how big it was I realized I had three options:

1. the naive approach, just add every coordinate to the CSV file
2. use a reverse geocoding service to get the country where the point belongs to
3. cluster set of points together

It was clear that the first approach wouldn’t have worked for two reasons, the former being that there were just too many points for OHM (the rendering is done on client side and that would slow things a lot). Also, I would just draw points onto a map without effectively creating a “heat map” so I discarded that option soon.
Using a reverse geocoding service wouldn’t have worked too: I should have ran too many requests to a service like this and it would have taken ages. Also, I would have ended up with per-country rather than a per-city highlighting and that would have faked the final result. So it was clear that the only viable option was to cluster set of points together and then produce a CSV file that OHM would understand. Soon I realized I needed some sort of spatial indexing for a 2d space that turned out to be quad trees.

Before we dive deeper in how to cluster the points together we need to understand what’s a quad-tree. In the classic, recursive, definition of a tree, a quad-tree is a tree where each node, that represent a coordinate point, has up to four children. Each child represent a relative position to its father, being north west, north east, south west or south east.

One requirement for generating the heat map is knowing how many nodes we clustered together. Thus it’s easy to define a node item as an object storing the coordinates of the point and the number of nodes that have been aggregated on that point. We can thus define a class like this (all the code examples following will be in Python):

1
2
3
4
5
6
7
8
9

class PointNode(object):
    NW, NE, SW, SE = 0, 1, 2, 3
    def __init__(self, lat, lon):
        self.lat, self.lon = float(lat), float(lon)
        self.nodes = [None] * 4
        self.aggregate_no = 1
 
    def __str__(self):
        return "%s, %s" % (self.lat, self.lon)

We’ll do two operations on the quad-tree: insert a new node and visit the whole tree. We can give a recursive definition of the insertion using the quad-tree as underlying data structure. Given a node:

1. if the tree’s node is empty just insert the node there and set the number of points clustered together for that node to 1
2. if the new node is near the tree’s node then compute a “middle node”, substitute it to the tree’s node and increment the number of points clustered together for that node
3. otherwise find out where the new node belongs in the quad-tree (north west, north east, south west or south east) and insert it there

The insert operation on the quad-tree can be thus coded like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def qtree_insert(root, node):
    """
    Insert a point into the quad tree substituting a node with its
    midpoint if the nodes are near to each other (less than DISTANCE_LIMIT)
    """
    if not root: return node
 
    # if we are under the distance limit, replace the root node with the
    # midpoint of the two nodes
    if point_distance(root, node) < DISTANCE_LIMIT:
        c = point_midpoint(root, node)
        c.nodes = root.nodes
        c.aggregate_no = root.aggregate_no + 1
        root = c
    else:
        # otherwise just insert the node where it belongs
 
        # exploit PointNode child indexing (with NW being 0 we just need to add
        # the proper number to get what we need)
        pos = PointNode.NW
        if node.lat > root.lat:
            pos += 2
        if node.lon > root.lon:
            pos += 1
 
        root.nodes[pos] = qtree_insert(root.nodes[pos], node)
    return root

The distance between two nodes can be computed using the pythagorean formula with parallel meridians. This formula returns the distance from two points in kilometers and it’s defined as: where is the Earth’s radius and , are two points coordinates in radians (thus and are the differences between the two points longitudes and latitudes).
Note that we can return the distance in miles or whatever distance measure we want just by converting the Earth’s radius accordingly. We should keep in mind though that this formula it not very accurate so if we need better accuracy we need to find a better alternative.

Browsing the whole tree can be done using a classic tree visit and, considering that for my purposes there’s no need to visit the nodes in a special order I chosen DFS to save some memory. The final source code of the Python script can be found on my repository on github. The whole process takes slightly more than one minute on my desktop machine. This, instead, is the final heat map that joined the This Week In Relevance contest:

I came back yesterday from the third Italian PyCon (aka pycon3) which was held in Florence and all I can say is that has been an amazing experience. I had the chance to meet a lot of new great people as well as the BDFL (which won’t be back in Europe for quite some time, as he said). Here follows a resume of what I think were the most interesting talks.

On first day, there were two keynotes: “A retrospective of how the community helped build Python 3.0”, held by Guido van Rossum and “Zen and the art of Abstractions’ maintenance” by Alex Martelli. I can just say that they were two extremely interesting talks which by the way weren’t diving too much — or any at all as in Guido’s talk — into code.

On the second day I really enjoyed two talks: “Erlang + Python, joining two worlds” by Lawrence Oluyede and a really great talk by Raymond Hettinger, “Easy AI with Python.” The former left me with a great curiosity about the functional languages world, while the latter really impressed me with how easy is to solve certain AI problems with Python (I solved many of the problems Raymond talked about previously, but never in Python and never really thought about even trying to).

On third day the Antonio Cangiano’s talk was enlightening. Even though it wasn’t really Python specific, he has given a great insight of how you can, well, “become rich with Python.”

Unfortunately I didn’t follow the Sunday afternoon’s talks since my airplane was leaving at 3.00pm, but at the end I can say that this was an incredible experience that I hope I can make again next year. And as a side note: the food was marvelous.

The last time I blogged about a new course I’m following at my university. This course, held by Pasquale Lops and Giovanni Semeraro, is very interesting at the point that I’ll be developing a custom information retrieval engine as part of my internship project. I can’t tell much more at this point since the internship haven’t started yet and I’m not sure I can release more details about this project (we’re still in the process of deciding if and how the whole thing will be released to the world).

In the meantime, I’ve been doing several experiments on this topic mostly about the memory usage and the performances of such system on limited hardware. This practically means implementing the algorithms you’ll be using and measuring the computational time they require.

One of the most common thing that our information retrieval engine have to do is to take a document and compress it, but considering that:

• this is a fundamental piece of this IR engine
• it will be used very often
• it’s not rare to process very large documents

You’ll get that this operation should be as efficient as possible.

I chosen to go down with zlib as my compression library for mainly two reasons:

• it’s already included in Python (this is not really a strong point since better compression algorithms are included in Python too)
• offers the best compromise in speed/compression ratio

Given the above considerations, let start coding our compression system.

We will use as our document example the PDF specifications, available at the Adobe Development Center (this is the file) that are 8.6Mb large.

So let start doing the things the basic way:

#!/usr/bin/env python
# compress1.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    out = ''
    for line in input_fd:
        out += cobj.compress(line)
    out += cobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    out = ''
    for line in input_fd:
        out += dobj.decompress(line)
    out += dobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

By running this program and performing a very basic profiling we get some indications:

kratorius@becks:~/compress$ time ./compress1.py compress PDF32000_2008.pdf compr.zlib
real    0m2.517s
user    0m1.496s
sys     0m0.060s

kratorius@becks:~/compress$ time ./compress1.py decompress compr.zlib decompr.pdf
real    0m0.640s
user    0m0.537s
sys     0m0.085s

We need 2.5 secs in order to compress a file smaller than 10Mb. This is quite unacceptable, since it means that we’re processing about 3.5Mb per second; so we need to understand what we’re doing wrong. I can spot at least two big errors in this script:

1. we’re reading the input file line by line that isn’t very efficient since in this way we’re accessing the disk multiple times (not counting that we are also processing the compression stuff line by line, that it’s not efficient and hasn’t so much sense in a binary file like our PDF)
2. we keep our compressed object in memory until we finish the compression, and this means that if the script would run faster, we’d still have a very high memory usage that is not optimal

So here it is the new version of our compression script that address the issues above:

#!/usr/bin/env python
# compress2.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.compress(input_fd.read(), compression_level)
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.decompress(input_fd.read())
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

Let perform our basic profiling again:

kratorius@becks:~/compress$ time ./compress2.py compress PDF32000_2008.pdf compr.zlib
real    0m1.668s
user    0m1.337s
sys     0m0.079s

kratorius@becks:~/compress$ time ./compress2.py decompress compr.zlib decompr.pdf
real    0m0.561s
user    0m0.394s
sys     0m0.086s

We are now reading the whole input file in memory (minimizing the disk accesses), compressing everything in memory and writing the compressed file to the output in a single shot. We got a high speedup in this way but we have just increased our memory usage since now we’re keeping in memory both the input and the compressed file. This could be optimal if we’re processing small files, but since we need to have a generalized approach, this solution is not that good.

We can do better. And we’ll do better in the third try:

#!/usr/bin/env python
# compress2.py
import zlib

READ_BYTES = 2097152 # 2Mb

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(cobj.compress(rd))

    output_fd.write(cobj.flush())

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(dobj.decompress(rd))

    output_fd.write(dobj.flush())

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

And we finally reached our goal:

kratorius@becks:~/compress$ time ./compress3.py compress PDF32000_2008.pdf compr.zlib
real    0m1.325s
user    0m1.226s
sys     0m0.070s

kratorius@becks:~/compress$ time ./compress3.py decompress compr.zlib decompr.pdf
real    0m0.534s
user    0m0.404s
sys     0m0.119s

This last try works because we’re still minimizing the disk accesses for small files (we’re reading 2Mb chunks per time) and this time we’re reducing the memory usage since:

• we read a 2Mb block from our input file
• we compress the read input
• we write it directly to our output file

I’m sure there’s still room for improvement but at this point we can be quite happy of our achievement. You can find the final script that performs error checking and file locking here (file locking works only on UNIX systems though, on Windows you should just comment the fcntl lines out). As always, suggestions are welcome.

It has been about a week since I began doing a deeper study of information retrieval. Actually, everything just began with a new course at my university about that and I just fallen in love almost immediately. The fact is that this thing really got me interested, and I began doing some experiments (one involves django as well, keep reading to know more).

In this week I learned a lot of things about information retrieval, text categorization, natural language processing and machine learning. But the most relevant thing is: the principles are easy, their implementation is not. The fact is that most of the techniques are relatively simple but you usually have to deal with very large datasets and this could be challenging, since one of the main requirements about information retrieval is time. It’s really much more important that you give less results in one second rather than giving better results in one hour. No one will ever care to use your system if it takes an hour to get some result. And if you’re considering to store your data in a database forget about normalization, it wouldn’t really take you anywhere.

Talking about storing informations, you know that if you’re dealing with documents most of the words are the so called stop words. Those stop words are words that doesn’t really mean anything, but they help the readers to get a better text flux. Classic examples of stop words are articles like “the”, “a”, “an” or logic connectors like “or” and “and”. These words are so common that their presence is quite useless since they’re are… everywhere. If you’re going to study information retrieval than you’ll learn about a weighting technique called tf-idf that gives a weight near to 0 to these words, but since you’d probably use a reverse index for words (an index that given a word, tells you in which documents that word appears) you can understand that this would take a lot of space if you’re going to include stop words.

So one of the biggest issues until now is that you’re going to deal with extremely large datasets, so you have to strip as many things as possible. Now consider those words: “fishing”, “fishes”, “fish”. They all talk about “fish”, and an user that is searching for “fish” would probably be interested in “fishes” or “fishing” as well. Additionally, it’s useless to store three words that are almost identical. So here comes the stemming that, by quoting the related wikipedia page, is the process for reducing inflected (or sometimes derived) words to their stem, base or root form – generally a written word form. Fortunately, if you’re dealing with english texts, there’s the Porter algorithm that is the state-of-the-art algorithm for this sort of things. But that works only with english, so if your documents are written in another language or they are written in multiple languages, things are going to be complicated.

This leads to think about the problem of the language identification. How do you know if some text is written in a language or in another just by looking at it? Of course you can describe the document’s language with some kind of meta tagging, but not all the documents have this kind of description, just think about the web. There are some kind of statistical methods based upon the classification of n-grams but I haven’t deeply investigated about them yet, so I can’t really say anything.

Now you got your collection of documents that match a certain query. Now: how do you know what document is more relevant than another (in other words: how do you rank pages)? You got two alternatives (well, probably more, but I know just these at this moment): the tf-idf that we said above and the cosine similarity. The latter is an interesting one: consider the tf-idf vectors of the documents, then consider the query as a document too. Now plot those tf-idf vectors and measure their cosine of the angle between them. The more you’re near to 1, the more relevant is the document.

There are a lot of other important things that need to be said like the precision and recall concept, but that’s enough for now. I’ll talk about this another time.

Anyway I’m doing an experimental project named django searchable. It’s a pluggable app for django that implements an information retrieval engine based on tf-idf weighting. Play with it if you’re brave enough.

Today I just ended one of my side projects: pytagram. Basically it generates an SVG file (that can successively be saved as eps/pdf/whatever and eventually manually manipulated) starting from a tree-like plain text file. This can be useful for generating cheat sheets or quick references to classes or functions that belongs to some project.

I did this for generating a django quick reference (here it is) since it has a lot of functions and I know what’s their purpose, but I can never remember the names (and now two A4 papers are right in front of me).

If you’re interested in this, check out the google code project page and grab your copy from the SVN repository.

There are tons of things that can be changed/optimized (i.e.: add some optional short explanation of the function, add more examples, easier way to change colors, …) but now the code is working quite well so that can be already useful to the people out there.

The decimal numeral system is composed of ten digits, which we represent as “0123456789” (the digits in a system are written from lowest to highest). Imagine you have discovered an alien numeral system composed of some number of digits, which may or may not be the same as those used in decimal. For example, if the alien numeral system were represented as “oF8”, then the numbers one through ten would be (F, 8, Fo, FF, F8, 8o, 8F, 88, Foo, FoF). We would like to be able to work with numbers in arbitrary alien systems. More generally, we want to be able to convert an arbitrary number that’s written in one alien system into a second alien system.

The above was exactly one of the practice problems of the google codejam (I still don’t know if I could join the event since I’ll probably be very busy with university exams in the day of the qualification round) and more generally the problem is the conversion of a number (that isn’t necessarily composed by the usual digits) from any base to any base. I just figured a solution out and passed both the the small input test and the large one. My solution is simple: convert the source base number in base 10 and then convert the produced base 10 number in another base. There are known algorithms for doing this (just think that 1986 for example is nothing but 1 * 10^3 + 9 * 10^2 + 8 * 10^1 + 6 * 10^0) and I finished implement my own solution in python.

The biggest issue here is that the source base can have symbols instead of digit and I solved this issue by mapping the symbols to an array and using the index value of the symbols as digit value. Here it is my solution:

#!/usr/bin/env python
import sys, array

def main(argv=None):
    if not argv:
        argv = sys.argv

    try:
        f = open(argv[1])
    except IOError:
        print "File doesn't exist"
        return 0

    try:
        i = 0
        for line in f:
            if i == 0:
                # first line
                line_num = int(line)
            else:
                number, input_b, output_b = line.strip('\n').split(' ')
                print 'Case #%d: %s' % (i, convert(number, input_b, output_b))

            i += 1
    finally:
        f.close()

    return 1

def convert(number, input_b, output_b):
    """
    Convert a number from any base to any base
    """

    return convert_from_10(convert_to_10(number, input_b), output_b)

def convert_to_10(input, base):
    """
    Input can be a number in any base, even in an 'alien' base.
    For example: 'Foo' could be a number in a numerical system
    whose digits are 'oF8'.

    Base is exactly the digits representation.
    If you want to convert that 'Foo' to base 10 then you must
    call ``convert_to_10('Foo', 'oF8')``.

    Remember that the number in ``base`` must be written in an
    ordered form

    Returns a string of the number in base 10
    """

    current_base = len(base)

    map_to_base = array.array('c')
    map(map_to_base.append, base)

    i = len(input) - 1
    base_10 = 0
    for digit in input:
        base_10 += map_to_base.index(digit) * current_base**i
        i -= 1

    return str(base_10)

def convert_from_10(input, base):
    """
    ``input`` is a number in base 10, while ``base`` is the digit
    representation of the new base (for example, for base 16 this
    could be '0123456789ABCDEF' or for an alien base 3 could be
    'oF8').

    Returns the number converted from base 10 to the specified
    base
    """

    map_to_base = array.array('c')
    map(map_to_base.append, base)

    current = int(input)
    base_n = ''
    while current != 0:
        base_n = map_to_base[current % len(base)] + base_n
        current = current / len(base)

    return base_n

if __name__ == '__main__':
    sys.exit(main())

And giving this input (the first line is number of the following lines):

4
9 0123456789 oF8
Foo oF8 0123456789
13 0123456789abcdef 01
CODE O!CDE? A?JM!.

I have the correct output:

Case #1: Foo
Case #2: 9
Case #3: 10011
Case #4: JAM!

Of course this is one of the practice problems and you should try to solve it by your own (otherwise it’s useless to try to join the event).

And I’ve been 6th. So I won a shared 2 hosting plan at webfaction and a 12 pack of G33K B33R caffeinated root beer (still trying to understand what this is exactly, anyway) from bawls. Anyway, here follows a short resume of what happened from Saturday through Tuesday (if you’re asking yourself why it didn’t ended on Sunday, well, keep reading).

The competition began very well, I worked normally for the first part of the day but then I had to stop for a while. When I came back, svn and djangodash website was not working anymore. I initially thought that it was some connection issue but when I saw that other sites were working properly so they definitely had some problems.

I just waited, then gone sleeping. In the morning I received in my mailbox a message that informed me of a big power outage in The Planet datacenter where webfaction hosts a lot of their server (among the which there was the djangodash one) caused by power generator’s explosion. Then the competition has been delayed for other two days, so I decided to take a breath and wait ‘till the svn would came back. But that didn’t happen on Sunday, so after a while I chosen (as the mail suggested) to work locally without committing anything at least until the svn return.

Then Monday came and I had other things to do, so I had to postpone djangodash for the evening when I’d freed myself from other, most urgent things. On Monday I did a very little coding, as well on Tuesday. So at the end of competition I cannot complete my project, and not even reach the 50% milestone.

Today I discovered that I was one of the winners (ok not really, 6th place was not really a good place, but at least I tried) and I really have to thanks the organizers for this event and hope to join another djangodash next year. Maybe, as I said to one of them in an email thread, hosting the site/svn in two different datacenters, just to be insured against eventual thunderstorms, tornado, earthquakes and so on…). I have to say that I really enjoyed the whole thing, and hope to have more competitors next year!

If you want to get more news about final process of djangodash with some stats, read this article on the Toast Driven website (that’s the company that ran the dash).

As probably many of you already knows, on May 31 will begin the Django dash competition. Djangodash is:

[…] is a chance for Django enthusiasts to flex their coding skills a little and put a fine point on “perfectionists with deadlines” by giving you a REAL deadline. 48 hours from start to stop to produce the best app you can and have a little fun in the process.

I’ll be participating, so if you haven’t registered yet, do it now! And don’t forget to check out the cool prizes :)

The Python’s built-in range() is an extremely useful function, but has a little problem: it doesn’t include the right extreme of the range. For example, a call to range(1, 10) will be evaluated to this a list of numbers from 1 to 9 (not including 10):

>>> range(1, 10)
[1, 2, 3, 4, 5, 6, 7, 8, 9]

Today I need for a work a range() function that includes the right extreme, so I had to develop mine. Here it is:

def inclusive_range(start, stop, step=1):
    """
    A range() clone, but this includes the extremes
    """
    l = []
    x = start
    while x <= stop:
        l.append(x)
        x += step
    return l

Of course there are faster implementations of this function around here (and if you know one, please let me know) and surely this one is not one of the fastest, but it works and that solves my problem right now.

Yesterday I had a meeting with a customer about a new site I should develop for them. Since they’re a book publisher, they wanted an online book store. Apart from the technical details (the site isn’t as simple as you may believe, they need a lot of not-so-easy-to-do stuff), the most important point we focused on is the fact that they have an internal IT technician that handles all their computer needs. If you’re asking yourself why this matters, keep reading:

• me (to be precise, my company) stopped development of PHP sites about one year ago in favor of Python
• we release the web site’s code to them
• for this project, we haven’t been asked any kind of future support; this means that when the site is finished, we won’t touch the product anymore (unless they don’t pay us to do the modifies they need)
• but they don’t want to pay us to these modifies, because they have their internal IT technician
• their technician knows only PHP (and he never even known the Python’s existence until yesterday)

So I had to illustrate why me and my company chosen Python for our web development needs, and here’s a summary of what I told them yesterday. Note that I’m not talking about why a language is better than the other, because this would move us in another direction, but why we chosen Python as our main programming language (even if this unconsciously lead us to say why, for us, Python is better than PHP, but that’s another story) :

• Time: remember that time is money. If I’d build an application in PHP, I’d spend about 1/3 of the time more if I’d develop the same application in Python
• Frameworks: nowadays all the popular languages have their web frameworks; Ruby has Ruby on Rails, PHP has Symfony, php.MVC, Python has Django, CherryPy, Pylons, and so on. But none of them (apart from rails, but we’re talking about PHP Vs Python) goes near to the completeness and functionality of django. And this take us at the previous point: less time in develop the same application
• Future modifies: even if for this project this isn’t the case, it’s much easier to modify a Python application than a PHP one due to the syntax of the language and its strong OOP orientation. You may argue that PHP 5 introduced a deep OOP support too, but that’s not the same thing and you know. PHP is born as procedural programming language, and even if we had OOP introduction in version 5, it doesn’t even comes near to Python under this point of view

With this I’m not saying that PHP is useless: what I mean is that Python is more convenient under the (our) business point of view. So if someone ask me why I use Python to do the same thing I can make with PHP, the answer will be: because with Python I can make the same thing in a shorter time and, consequently, with a lower budget.