Recently I needed to show a heat map of a quite a lot of coordinate points for a little project of mine that ended up in a data visualization contest (that unfortunately I didn’t win, even though I made to the finalists). The idea was to show the distribution of the georeferenced wikipedia pages through a heat map, so when I first heard about openheatmap.com I knew it was the tool to use. OpenHeatMap.com is an excellent project by Pete Warden that takes a dataset as a CSV, Excel or Google Spreadsheet file and convert it to a nice, browsable heat map presentation.

The first step was to obtain I dataset I could work on. I first tried to work directly onto the whole wikipedia database dump, extracting all the georeferenced pages in a smaller dataset. I actually succeeded but this work included only the english georeferenced pages. Also, extracting and converting coordinates to a common format would have been a real pain. So instead I decided to use the dump from the wikipedia-world project that already included data in a CSV file from all the downloadable wikipedia dumps which include other languages other than english. This dataset include roughly 1.300.000 points, so I had to narrow down some options to process it.

Once I had the dataset ready and knew how big it was I realized I had three options:

1. the naive approach, just add every coordinate to the CSV file
2. use a reverse geocoding service to get the country where the point belongs to
3. cluster set of points together

It was clear that the first approach wouldn’t have worked for two reasons, the former being that there were just too many points for OHM (the rendering is done on client side and that would slow things a lot). Also, I would just draw points onto a map without effectively creating a “heat map” so I discarded that option soon.
Using a reverse geocoding service wouldn’t have worked too: I should have ran too many requests to a service like this and it would have taken ages. Also, I would have ended up with per-country rather than a per-city highlighting and that would have faked the final result. So it was clear that the only viable option was to cluster set of points together and then produce a CSV file that OHM would understand. Soon I realized I needed some sort of spatial indexing for a 2d space that turned out to be quad trees.

Before we dive deeper in how to cluster the points together we need to understand what’s a quad-tree. In the classic, recursive, definition of a tree, a quad-tree is a tree where each node, that represent a coordinate point, has up to four children. Each child represent a relative position to its father, being north west, north east, south west or south east.

One requirement for generating the heat map is knowing how many nodes we clustered together. Thus it’s easy to define a node item as an object storing the coordinates of the point and the number of nodes that have been aggregated on that point. We can thus define a class like this (all the code examples following will be in Python):

1
2
3
4
5
6
7
8
9

class PointNode(object):
    NW, NE, SW, SE = 0, 1, 2, 3
    def __init__(self, lat, lon):
        self.lat, self.lon = float(lat), float(lon)
        self.nodes = [None] * 4
        self.aggregate_no = 1
 
    def __str__(self):
        return "%s, %s" % (self.lat, self.lon)

We’ll do two operations on the quad-tree: insert a new node and visit the whole tree. We can give a recursive definition of the insertion using the quad-tree as underlying data structure. Given a node:

1. if the tree’s node is empty just insert the node there and set the number of points clustered together for that node to 1
2. if the new node is near the tree’s node then compute a “middle node”, substitute it to the tree’s node and increment the number of points clustered together for that node
3. otherwise find out where the new node belongs in the quad-tree (north west, north east, south west or south east) and insert it there

The insert operation on the quad-tree can be thus coded like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

def qtree_insert(root, node):
    """
    Insert a point into the quad tree substituting a node with its
    midpoint if the nodes are near to each other (less than DISTANCE_LIMIT)
    """
    if not root: return node
 
    # if we are under the distance limit, replace the root node with the
    # midpoint of the two nodes
    if point_distance(root, node) < DISTANCE_LIMIT:
        c = point_midpoint(root, node)
        c.nodes = root.nodes
        c.aggregate_no = root.aggregate_no + 1
        root = c
    else:
        # otherwise just insert the node where it belongs
 
        # exploit PointNode child indexing (with NW being 0 we just need to add
        # the proper number to get what we need)
        pos = PointNode.NW
        if node.lat > root.lat:
            pos += 2
        if node.lon > root.lon:
            pos += 1
 
        root.nodes[pos] = qtree_insert(root.nodes[pos], node)
    return root

The distance between two nodes can be computed using the pythagorean formula with parallel meridians. This formula returns the distance from two points in kilometers and it’s defined as: where is the Earth’s radius and , are two points coordinates in radians (thus and are the differences between the two points longitudes and latitudes).
Note that we can return the distance in miles or whatever distance measure we want just by converting the Earth’s radius accordingly. We should keep in mind though that this formula it not very accurate so if we need better accuracy we need to find a better alternative.

Browsing the whole tree can be done using a classic tree visit and, considering that for my purposes there’s no need to visit the nodes in a special order I chosen DFS to save some memory. The final source code of the Python script can be found on my repository on github. The whole process takes slightly more than one minute on my desktop machine. This, instead, is the final heat map that joined the This Week In Relevance contest:

For a problem I’m working on I got stuck onto the classical situation of local maximum. After trying to work around the problem in several more or less creative ways, I thought of the simulated annealing algorithm. Considering it’s been a while since I last saw it I tried to search for it on the web and surprisingly there is not much stuff about it, and the few bits I found are often contraddictory. After quite a lot of digging I decided to write about it here. As a warning I should probably say that there will be digging into some basic statistic and complexity analysis, as well as a quick formal introduction to the problem of the knapsack. You should be able to follow even if you don’t know nothing about those topics, but having some foundations in these areas would be of great help.

Let begin with the knapsack problem. This is a classic combinatorial computer science problem known to be NP-complete, meaning that the exact optimal solution cannot be found in polynomial time. This often means that most of the times we are happy of a good solution, assuming it’s not so far from the optimal one. In the simpliest possible terms you are a thief and you’re in a room with a set of objects that are worth something but you have only one knapsack, and that knapsack can carry at most a certain weight, so you have to choose carefully what objects to steal in order to maximize the earnings. For example consider the following situation: you can carry at most 5kg, and there is one laptop and a 4kg safe with pure diamonds within. You can’t carry both of them so you have to choose what’s better to carry on, the diamonds or the laptop. A smart thief would choose the diamonds since their value is considerably higher than the laptop.

There are few variations of the same problem but most common one is named “0-1”: you can’t split the weight over two or more carriers or bags but either you take the whole weight or you leave the object where it is. Mathematically talking, consider a set of objects, each item is worth and weights with . Then the goal is to maximize the following function:

But keeping the following constraint (being the maximum weight we can carry):

The first problem we have to face then is how we should generate the items and how the value of a solution should be calculated. The following example, as the other that will follow, it’s written in Python but it’s quite easy to understand so porting to another language wouldn’t be that hard. I chosen to generate 50 objects with values that range from 1 to 99$ (both ends included) using a uniform distribution (if you don’t know much about statistic, it means that all the values are equally distributed among the objects). The same with the weights except they range from 1 to 20 (the choice of the weight’s unit measure is left to you).

def generate_items(n_items=100):
    "Generate a list of items that could be stealed"
    items = []
    for n in range(0, n_items):
        # use a uniform distribution both for values and for weights
        cost = random.randint(1, 100)
        weight = random.uniform(1, 20)
        items.append((cost, weight))
    return items

So the items are nothing other than a list of pairs in the format for every object . Probably a better and more realistic dataset would have used a normal distribution for the weights but it’s trivial to change the generation to function to work in that way. For our purposes the uniform distribution does its job.

As we said above the problem is NP-complete so we usually need to visit the whole search space to get the optimal solution which can be quite big when as the number of objects grows. Here comes the simulated annealing. We won’t visit the search space extensively, but we’d rather generate solutions. Indeed, it is a stochastic heuristic search algorithm. An heuristic is a function that measures how much something is good or bad, and stochastic means that we move more or less in a random way into the search space. In practical terms it’s not greedy as that it doesn’t always follow what the heuristic says but rather randomly search where the heuristic function points to. For example consider the needle in the haystack situation: an exaustive search method would take every straw piece, check that it’s not a needle, put it apart and repeat those moves until you don’t find the needle. You don’t want to proceed in this way, you’re more likely to end in less time if you look randomly in the haystack and if somethings stings you while you’re holding straw then search into that straw piece, because there may be the needle in there. In the knapsack problem we do have the heuristic, and it’s the function above that, for each solution (admissible or not), says how much it’s worth. In this case we define an admissible solution as one that it’s not too heavy.

An useful (and classic as well) example is the one of the blind hill climbing. You’re blind and stuck on a hill and you need to reach the top. A good principle that could lead you to the top is to touch the terrain and always follow the rising path. It could, because if you’re on a rock than you surely haven’t reached the top but the principle above doesn’t apply: you reached a local maximum (invert the things and you get the same thing for a local minimum). Simulated annealing avoids these problems by trying worsening moves from time to time: even if this may not sound like a good move it helps avoiding the problems we described above.

In the function above once in the middle we could choose to take the left maximum (which is a local maximum). Using hill climbing we’d be stuck on that because we wouldn’t try other paths.

Simulated annealing takes its name from the same process that metals go through when cooling from a melting point. Indeed, the cooling process consists of several particles that changes energy states (this statement may not be accurate or be inexact at all, but please forgive me as I never studied those things and I all know in this field comes from simulated annealing algorithm itself), in particular we can calculate the transiction probability from one state to another. Considering two energy states and and a temperature , switching from to has probability:

Where is a constant called Boltzmann’s constant. But then, how do we apply those statements to our problem (or a combinatorial search problem, in general)? The most important concept to grasp is the energy switching one. As a particle change state, a solution might change. Indeed for the knapsack problem there are many admissible solution, each one with an associated earning. Of course we’d prefer the one with the higher earnings (and simulated annealing will help us find that) but still it’s perfectly acceptable to go from a solution to another as long as the other solution continues to be admissible.

So here it is what the simulated annealing does: if you find a better item go on an take that path (under this circumnstance, behaves just like the hill climbing), otherwise change state with probability (you may notice that the Boltzmann’s constant is missing, indeed that constant applies mostly to thermodynamic when dealing with different metals). The effect that the temperature scaling has is that at higher temperatures it’ll try worsening moves quite often while on lower temperatures the probability to do worsening moves is lower so when temperature tends towards 0 it behaves quite like the hill climbing algorithm.

def simulated_annealing(solution, items, max_weight):
    "Apply the simulated annealing for solving the knapsack problem"
    best = solution
    best_value = compute_cost(solution, items)[0]
    current_sol = solution
    temperature = 1.0
 
    while True:
        current_value = compute_cost(best, items)[0]
 
        for i in range(0, COOLING_STEPS):
            moves = generate_moves(current_sol, items, max_weight)
            idx = random.randint(0, len(moves) - 1)
            random_move = moves[idx]
 
            delta = compute_cost(random_move, items)[0] - compute_cost(best, items)[0]
 
            if delta > 0:
                best = random_move
                best_value = compute_cost(best, items)[0]
                current_sol = random_move
            else:
                if math.exp(delta / float(temperature)) > random.random():
                    current_sol = random_move
 
        temperature = TEMP_ALPHA * temperature
        if current_value >= best_value or temperature <= 0:
            break

And finally, that is the simulated annealing. You start from a temperature of 1.0 then you have a certain number of cooling steps, in every one of them you extract a random item from the neighbours and, if the item is better than the current best item then it becomes the new best item (and the new local solution). If the new item’s value is worst than the current best then update the current local solution with the probability expressed above. After the cooling steps the temperature is decreased with an exponential decay (usually it is ).

In the example above you don’t wait for the temperature to be 0 but you leave the loop if after all the cooling steps there hasn’t been any improvement. How big must be the number of cooling steps and the value it’s a fine tuning problem. A different approach that allow to get rid of the cooling steps is to make the temperature get cold slowly ( and is a very small value like 0.01). Besides, a common improvement is to cache the moves within the cooling steps unless you found a new best value or changed state.

You can see that the most critical points are the neighbour generation and the cost computation. While the neighbour generation could be cached like I said above, the cost computation could be replaced with a probability estimate in order to reduce the time per cooling step.

But how do you apply the algorithm? If an empty solution is acceptable then you can just start with that and let the neighbour’s generator to create a solution, but usually you start with a greedy solution (found through the hill climbing, for example) or from a random one.

Here follows the complete code. I used 1000 cooling steps and a value of 0.8. I start from a random solution whose behaviour is not that bad considering how much time it takes to compute the solution. Indeed often random algorithms perform really well on combinatorial algorithms, see stochastic search for some more informations.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120

#!/usr/bin/env python
import math
import operator
import pprint
import random
import sys
 
COOLING_STEPS = 1000
TEMP_ALPHA = 0.8
 
random.seed()
 
def generate_items(n_items=100):
    "Generate a list of items that could be stealed"
    items = []
    for n in range(0, n_items):
        # use a uniform distribution both for values and for weights
        cost = random.randint(1, 100)
        weight = random.uniform(1, 20)
        items.append((cost, weight))
    return items
 
def main(args):
    items = generate_items()
    pprint.pprint(items)
 
    start_sol = generate_random_solution(items, max_weight=40)
    print "Random solution: %s" % start_sol
    print "value: (cost: %d, weight: %f)" % compute_cost(start_sol, items)
 
    solution = simulated_annealing(start_sol, items, max_weight=40)
    print "Final solution: %s" % solution
    print "value: (cost: %d, weight: %f)" % compute_cost(solution, items)
 
    return False
 
def generate_random_solution(items, max_weight):
    "Generate a starting random solution"
 
    # generate a random solution by adding a random item
    # until we don't get over the weight
    solution = []
    while compute_cost(solution, items)[1] <= max_weight:
        idx = random.randint(0, len(items) - 1)
        # skip duplicates
        if idx not in solution:
            solution.append(idx)
    # last item makes us get over the weight so simply remove it
    # we'll look for better results after
    solution = solution[:-1]
    return solution
 
def simulated_annealing(solution, items, max_weight):
    "Apply the simulated annealing for solving the knapsack problem"
    best = solution
    best_value = compute_cost(solution, items)[0]
    current_sol = solution
    temperature = 1.0
 
    while True:
        current_value = compute_cost(best, items)[0]
 
        for i in range(0, COOLING_STEPS):
            moves = generate_moves(current_sol, items, max_weight)
            idx = random.randint(0, len(moves) - 1)
            random_move = moves[idx]
 
            delta = compute_cost(random_move, items)[0] - \
                    compute_cost(best, items)[0]
 
            if delta > 0:
                best = random_move
                best_value = compute_cost(best, items)[0]
                current_sol = random_move
            else:
                if math.exp(delta / float(temperature)) > random.random():
                    current_sol = random_move
 
        temperature = TEMP_ALPHA * temperature
        if current_value >= best_value or temperature <= 0:
            break
    return best
 
def generate_moves(solution, items, max_weight):
    """
    Generate all the ammissible moves starting from the input
    solution
    """
    moves = []
    # try to add another item and save as a possible move
    for idx, item in enumerate(items):
        if idx not in solution:
            move = solution[::]
            move.append(idx)
 
            if compute_cost(move, items)[1] <= max_weight:
                moves.append(move)
 
    # try to remove one item
    for idx, item in enumerate(solution):
        move = solution[::]
        del move[idx]
        if move not in moves:
            moves.append(move)
 
    return moves
 
def compute_cost(solution, items):
    """
    Return a tuple in the format (id_item1, id_item2, ...)
    for the input solution
    """
    cost, weight = 0, 0
    for item in solution:
        cost += items[item][0]
        weight += items[item][1]
    return (cost, weight)
 
if __name__ == '__main__':
    sys.exit(main(sys.argv))

The results are suprising. For three different set of 100 items, each one with its own value and weight, here are the results:

$ python sa.py
Random solution: [98, 71, 95]
value: (cost: 44, weight: 27.001685)
Final solution: [71, 95, 67, 9, 41, 33, 27]
value: (cost: 229, weight: 39.791386)

$ python sa.py
Random solution: [38, 16, 62, 31]
value: (cost: 124, weight: 36.863846)
Final solution: [38, 16, 62, 31, 5]
value: (cost: 194, weight: 38.970745)

Random solution: [61, 44, 48, 38]
value: (cost: 293, weight: 30.357135)
Final solution: [61, 44, 48, 38, 37, 5, 2]
value: (cost: 421, weight: 39.331549)

We usually don’t leave much free weight and the quality of the solutions found is quite good considering that the time required to compute that is ~0.5s.

A couple of days ago I finally had my Google Wave sandbox account. Given that I just finished developing my very first robot, I thought I’d share some impressions on the whole thing. From the user-side, things are far from being ready. Some important features are still missing, just to name one you can’t remove user from a wave once they joined (or, alternatively, there’s no way to ignore a wave). Indeed, given that I joined several waves to try other people applications, I’m getting continuous notifications. But anyway, the whole thing is to me like a great development playground where I can make all the sorts of experiments.

They’ve been loyal: when you signed up the registration form, they asked you if you were comfortable with APIs changing or an instable system. That’s what you’ll find once you get your sandbox account. APIs are there but haven’t been fully documented yet and most of your work when developing some robot/gadget will be in exploring the API sources (they’re open source, yau!) or searching for some examples on the samples gallery, which is an invaluable resource by the way.

Debugging is hard too, given that you can’t test what you done locally but you have to upload your code to AppEngine to see if it works (actually AppEngine is the only platform they accept requests from, but they plan to allow every host that talks the wave protocol in the future). This means that if, for example, there’s some typo in the code (i.e.: appendText() rather than AppendText()), you’d know only by looking at the AppEngine logs.

Be prepared to experience casual failures too. Sometimes your robot is working correctly and is receiving the whole wavelet (which is the whole conversation thread), but its response is ignored by the server for some unknown cause.

Anyway, even though there’s still some clear work in progress, I felt like the whole thing was quite exciting both from the user and as the developer point of view. The event model they thought for the external applications perfectly fits the nature of The Wave and gives room for some nice asynchronous applications. Hopefully, we’ll meet on Google Wave soon.

One of the reasons I haven’t been writing on this blog that much lately is that I’ve been terribly busy with university given that I just cleared out six exams in six months. That said, for one of my three exams that I still have left, I had to develop an inference engine written in C++. Since this was a fairly large project that had to deal with some sort of NP-complete problems (see also: unification) and given that this was the first time I wrote something serious in C++ (i.e.: that would involve more than a class and that didn’t contain the “Hello world” string) I had the chance to learn quite a few new things.

First thing, no matter how good your data structure is or how well you implemented that, sooner or later you’ll meet a speed barrier that even the best data structure for the job can’t beat. That doesn’t mean you don’t have to think about what data structure to use, besides I’m really passionate about finding the right data structure for the job so if I have the chance to deal with this kind of problems seems like I won the lottery. Sometimes the algorithm you’re using simply has some limits that can’t be beaten unless you radically adjust your algorithm or you completely change it with some a better-algorithm-for-the-job. Say we have an implicit, non-weighted graph, but we know that at most we’d be expanding fifty nodes of about a hundred bytes each and the expansion of every node comes cheap as computational cost. Now you want to find a particular node within that graph and along with it you want to find the shortest path to the starting node. Given these considerations, what algorithm would you use in this case? I’d go for breadth-first search, since we don’t lose much time expanding the nodes (computational cost of each node expansion), we have relatively few nodes that, unless we’re running on some special hardware that is somewhat limited (even though today even those automatized toilet’s chips can hold 5Kb in memory), will take a very little amount of memory.

Now suppose we still have that implicit graph, with our fifty nodes of about a hundred bytes each. But now we know that generating every node is very expensive in terms of computational costs. The BFS above could still be applicable under certain circumstances but we’d better be looking for alternatives. You can hold the graph in whatever data structure you want but unless you decide to change the algorithm with something better there’s nothing you can do about. Of course you can gain some speed by improving your data structure but that’s not the point since what makes the shortest path search slow is the generation of a new node. So what you have to do? Try to generate as few nodes as possible. Indeed, say our nearest solution is ten edges far from the starting node, with BFS you’d expand first all the nodes which have a direct connection to the starting node (so one edge), than from each of these nodes you’d be expanding other nodes that now will have two edges separating them from the starting nodes, and so on until you expand all the levels and get to the solution which is separated from the starting node by ten edges. You had the solution eventually, but it came at an expensive price. What are the alternatives then? Say we know, for each node we expand, a numerical value which says how good that node is in terms of distance from what we’re looking for. In this way we can follow only the good leads and leaving the bad paths out of our research (that, though, is not entirely true, and I’ll say why in a minute) and still, we have limited the number of expanded nodes and as result we had a great speed up. There’s an entire category of algorithms that are based upon the principle that you know something of your problem that can help you out in some cases and these are called informed algorithms. One of those who comes to my mind is A* which is quite simple to implement too. These algorithms are based upon the assumption that you can make some estimation of how good the current state is. Indeed, most of these algorithm’s accuracy comes from how good the function that gives you that estimation is. This function is called the heuristic function. But the heuristic function is an estimation and it’s likely to be wrong in some cases. So you still end up expanding more nodes than necessary and following some wrong leads, but now you save yourself from exploring all the wrong leads that the previous BFS would have forced you to do.

For the project I was talking above, at a certain point we hit the limit. We tried switching data structures initially and that saved some time but still we weren’t able to come up with any consistent time reduction. Until we switched algorithm, then we gained something like a 80% speed up and we even had the chance to use some simpler data structure which allowed us to exploit some stuff like caching which I suppose is a large part of that 80%.

Saying that, here comes the second lesson: know your environment. You have to know how your processor works, what’s the difference between, say, a L2 and a L1 cache, why disk access is slow and how it works, how the paging works and so on. These things can make great room for optimization if you know how to use them. Caching, for example, is probably one a thing that may give a great help to you. If you know what the cache line is and how big is on your processor, you’ll know how and when you can exploit the data locality principle. Knowing that disk reading is not that slow as it’s said could really help you (I heard you crying: I did not mistyped, disk read ain’t slow, what’s makes this operation slow is the time the disk’s head takes to position itself. Once we got the head in position, reading a whole block is probably faster than you imagine. Of course, I’m talking about old-style mechanical disks, with these shiny new solid state disks is another story). In the same way it can help knowing that your program will not cause many page faults or won’t fragment the allocation memory because your data structure doesn’t fit in one page or your algorithm makes thousand of allocations/deallocations of different size. This is very important, even though I got to admit is one of the topics I’m lagging behind.

Then you chosen the best algorithm in the world, you resurrected the dead in order to fit your data structure in exactly one page and you used condoms while you were coding, but still your program is very slow and you still can’t find the reason. Well I tell you, probably you’re using the wrong data structure but at this point I guess you’d know. With the project I said above, we experienced exactly this. We did everything we can to speed up the algorithms, we optimized everything could have been optimized but yet our program was slow until we realized that something was a real bottleneck for our purpose. Big Oh analysis it’s really useful, but you got to take into account that that little ‘n’ in that big ‘o’ is meant to be big. Indeed, that gives you an idea of how the data structure works as n grows, so that two same algorithms or data structures which both have a O(nlogn) complexity can be very different. In our case, the bottleneck was the STL map implementation. It turns out that, under the hood, it is a red-black tree that was everything but fast in our program. We were spending 30% of the time within the STL map looking for the key. The problem here really was that we hadn’t many values within the map, most I’ve seen were ten items but we had an average of three or four items. When we switched from the usual map to a very simple hash, we had a terrible speed improvement.

Next thing is optimizing little stuff. This won’t yield greet speed ups but yet, they’re useful. We gained about one second by only switching one function’s calling convention to fastcall and we had even greater benefits by forcing the inlining of some other functions.

Other than that, having to work on a large C++ code base has been challenging, most because of some C++ gotchas (i.e.: why something like string fn() {} returns NULL implicitly?) even though I come from a assembler/C background and I have a solid OO theory base behind. In the end, though, it has been probably the funniest problem I faced in the latest years and will be eventually be open sourced after the summer.

The last time I blogged about a new course I’m following at my university. This course, held by Pasquale Lops and Giovanni Semeraro, is very interesting at the point that I’ll be developing a custom information retrieval engine as part of my internship project. I can’t tell much more at this point since the internship haven’t started yet and I’m not sure I can release more details about this project (we’re still in the process of deciding if and how the whole thing will be released to the world).

In the meantime, I’ve been doing several experiments on this topic mostly about the memory usage and the performances of such system on limited hardware. This practically means implementing the algorithms you’ll be using and measuring the computational time they require.

One of the most common thing that our information retrieval engine have to do is to take a document and compress it, but considering that:

• this is a fundamental piece of this IR engine
• it will be used very often
• it’s not rare to process very large documents

You’ll get that this operation should be as efficient as possible.

I chosen to go down with zlib as my compression library for mainly two reasons:

• it’s already included in Python (this is not really a strong point since better compression algorithms are included in Python too)
• offers the best compromise in speed/compression ratio

Given the above considerations, let start coding our compression system.

We will use as our document example the PDF specifications, available at the Adobe Development Center (this is the file) that are 8.6Mb large.

So let start doing the things the basic way:

#!/usr/bin/env python
# compress1.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    out = ''
    for line in input_fd:
        out += cobj.compress(line)
    out += cobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    out = ''
    for line in input_fd:
        out += dobj.decompress(line)
    out += dobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

By running this program and performing a very basic profiling we get some indications:

kratorius@becks:~/compress$ time ./compress1.py compress PDF32000_2008.pdf compr.zlib
real    0m2.517s
user    0m1.496s
sys     0m0.060s

kratorius@becks:~/compress$ time ./compress1.py decompress compr.zlib decompr.pdf
real    0m0.640s
user    0m0.537s
sys     0m0.085s

We need 2.5 secs in order to compress a file smaller than 10Mb. This is quite unacceptable, since it means that we’re processing about 3.5Mb per second; so we need to understand what we’re doing wrong. I can spot at least two big errors in this script:

1. we’re reading the input file line by line that isn’t very efficient since in this way we’re accessing the disk multiple times (not counting that we are also processing the compression stuff line by line, that it’s not efficient and hasn’t so much sense in a binary file like our PDF)
2. we keep our compressed object in memory until we finish the compression, and this means that if the script would run faster, we’d still have a very high memory usage that is not optimal

So here it is the new version of our compression script that address the issues above:

#!/usr/bin/env python
# compress2.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.compress(input_fd.read(), compression_level)
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.decompress(input_fd.read())
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

Let perform our basic profiling again:

kratorius@becks:~/compress$ time ./compress2.py compress PDF32000_2008.pdf compr.zlib
real    0m1.668s
user    0m1.337s
sys     0m0.079s

kratorius@becks:~/compress$ time ./compress2.py decompress compr.zlib decompr.pdf
real    0m0.561s
user    0m0.394s
sys     0m0.086s

We are now reading the whole input file in memory (minimizing the disk accesses), compressing everything in memory and writing the compressed file to the output in a single shot. We got a high speedup in this way but we have just increased our memory usage since now we’re keeping in memory both the input and the compressed file. This could be optimal if we’re processing small files, but since we need to have a generalized approach, this solution is not that good.

We can do better. And we’ll do better in the third try:

#!/usr/bin/env python
# compress2.py
import zlib

READ_BYTES = 2097152 # 2Mb

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(cobj.compress(rd))

    output_fd.write(cobj.flush())

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(dobj.decompress(rd))

    output_fd.write(dobj.flush())

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

And we finally reached our goal:

kratorius@becks:~/compress$ time ./compress3.py compress PDF32000_2008.pdf compr.zlib
real    0m1.325s
user    0m1.226s
sys     0m0.070s

kratorius@becks:~/compress$ time ./compress3.py decompress compr.zlib decompr.pdf
real    0m0.534s
user    0m0.404s
sys     0m0.119s

This last try works because we’re still minimizing the disk accesses for small files (we’re reading 2Mb chunks per time) and this time we’re reducing the memory usage since:

• we read a 2Mb block from our input file
• we compress the read input
• we write it directly to our output file

I’m sure there’s still room for improvement but at this point we can be quite happy of our achievement. You can find the final script that performs error checking and file locking here (file locking works only on UNIX systems though, on Windows you should just comment the fcntl lines out). As always, suggestions are welcome.

It has been about a week since I began doing a deeper study of information retrieval. Actually, everything just began with a new course at my university about that and I just fallen in love almost immediately. The fact is that this thing really got me interested, and I began doing some experiments (one involves django as well, keep reading to know more).

In this week I learned a lot of things about information retrieval, text categorization, natural language processing and machine learning. But the most relevant thing is: the principles are easy, their implementation is not. The fact is that most of the techniques are relatively simple but you usually have to deal with very large datasets and this could be challenging, since one of the main requirements about information retrieval is time. It’s really much more important that you give less results in one second rather than giving better results in one hour. No one will ever care to use your system if it takes an hour to get some result. And if you’re considering to store your data in a database forget about normalization, it wouldn’t really take you anywhere.

Talking about storing informations, you know that if you’re dealing with documents most of the words are the so called stop words. Those stop words are words that doesn’t really mean anything, but they help the readers to get a better text flux. Classic examples of stop words are articles like “the”, “a”, “an” or logic connectors like “or” and “and”. These words are so common that their presence is quite useless since they’re are… everywhere. If you’re going to study information retrieval than you’ll learn about a weighting technique called tf-idf that gives a weight near to 0 to these words, but since you’d probably use a reverse index for words (an index that given a word, tells you in which documents that word appears) you can understand that this would take a lot of space if you’re going to include stop words.

So one of the biggest issues until now is that you’re going to deal with extremely large datasets, so you have to strip as many things as possible. Now consider those words: “fishing”, “fishes”, “fish”. They all talk about “fish”, and an user that is searching for “fish” would probably be interested in “fishes” or “fishing” as well. Additionally, it’s useless to store three words that are almost identical. So here comes the stemming that, by quoting the related wikipedia page, is the process for reducing inflected (or sometimes derived) words to their stem, base or root form – generally a written word form. Fortunately, if you’re dealing with english texts, there’s the Porter algorithm that is the state-of-the-art algorithm for this sort of things. But that works only with english, so if your documents are written in another language or they are written in multiple languages, things are going to be complicated.

This leads to think about the problem of the language identification. How do you know if some text is written in a language or in another just by looking at it? Of course you can describe the document’s language with some kind of meta tagging, but not all the documents have this kind of description, just think about the web. There are some kind of statistical methods based upon the classification of n-grams but I haven’t deeply investigated about them yet, so I can’t really say anything.

Now you got your collection of documents that match a certain query. Now: how do you know what document is more relevant than another (in other words: how do you rank pages)? You got two alternatives (well, probably more, but I know just these at this moment): the tf-idf that we said above and the cosine similarity. The latter is an interesting one: consider the tf-idf vectors of the documents, then consider the query as a document too. Now plot those tf-idf vectors and measure their cosine of the angle between them. The more you’re near to 1, the more relevant is the document.

There are a lot of other important things that need to be said like the precision and recall concept, but that’s enough for now. I’ll talk about this another time.

Anyway I’m doing an experimental project named django searchable. It’s a pluggable app for django that implements an information retrieval engine based on tf-idf weighting. Play with it if you’re brave enough.

Today I just ended one of my side projects: pytagram. Basically it generates an SVG file (that can successively be saved as eps/pdf/whatever and eventually manually manipulated) starting from a tree-like plain text file. This can be useful for generating cheat sheets or quick references to classes or functions that belongs to some project.

I did this for generating a django quick reference (here it is) since it has a lot of functions and I know what’s their purpose, but I can never remember the names (and now two A4 papers are right in front of me).

If you’re interested in this, check out the google code project page and grab your copy from the SVN repository.

There are tons of things that can be changed/optimized (i.e.: add some optional short explanation of the function, add more examples, easier way to change colors, …) but now the code is working quite well so that can be already useful to the people out there.

The decimal numeral system is composed of ten digits, which we represent as “0123456789” (the digits in a system are written from lowest to highest). Imagine you have discovered an alien numeral system composed of some number of digits, which may or may not be the same as those used in decimal. For example, if the alien numeral system were represented as “oF8”, then the numbers one through ten would be (F, 8, Fo, FF, F8, 8o, 8F, 88, Foo, FoF). We would like to be able to work with numbers in arbitrary alien systems. More generally, we want to be able to convert an arbitrary number that’s written in one alien system into a second alien system.

The above was exactly one of the practice problems of the google codejam (I still don’t know if I could join the event since I’ll probably be very busy with university exams in the day of the qualification round) and more generally the problem is the conversion of a number (that isn’t necessarily composed by the usual digits) from any base to any base. I just figured a solution out and passed both the the small input test and the large one. My solution is simple: convert the source base number in base 10 and then convert the produced base 10 number in another base. There are known algorithms for doing this (just think that 1986 for example is nothing but 1 * 10^3 + 9 * 10^2 + 8 * 10^1 + 6 * 10^0) and I finished implement my own solution in python.

The biggest issue here is that the source base can have symbols instead of digit and I solved this issue by mapping the symbols to an array and using the index value of the symbols as digit value. Here it is my solution:

#!/usr/bin/env python
import sys, array

def main(argv=None):
    if not argv:
        argv = sys.argv

    try:
        f = open(argv[1])
    except IOError:
        print "File doesn't exist"
        return 0

    try:
        i = 0
        for line in f:
            if i == 0:
                # first line
                line_num = int(line)
            else:
                number, input_b, output_b = line.strip('\n').split(' ')
                print 'Case #%d: %s' % (i, convert(number, input_b, output_b))

            i += 1
    finally:
        f.close()

    return 1

def convert(number, input_b, output_b):
    """
    Convert a number from any base to any base
    """

    return convert_from_10(convert_to_10(number, input_b), output_b)

def convert_to_10(input, base):
    """
    Input can be a number in any base, even in an 'alien' base.
    For example: 'Foo' could be a number in a numerical system
    whose digits are 'oF8'.

    Base is exactly the digits representation.
    If you want to convert that 'Foo' to base 10 then you must
    call ``convert_to_10('Foo', 'oF8')``.

    Remember that the number in ``base`` must be written in an
    ordered form

    Returns a string of the number in base 10
    """

    current_base = len(base)

    map_to_base = array.array('c')
    map(map_to_base.append, base)

    i = len(input) - 1
    base_10 = 0
    for digit in input:
        base_10 += map_to_base.index(digit) * current_base**i
        i -= 1

    return str(base_10)

def convert_from_10(input, base):
    """
    ``input`` is a number in base 10, while ``base`` is the digit
    representation of the new base (for example, for base 16 this
    could be '0123456789ABCDEF' or for an alien base 3 could be
    'oF8').

    Returns the number converted from base 10 to the specified
    base
    """

    map_to_base = array.array('c')
    map(map_to_base.append, base)

    current = int(input)
    base_n = ''
    while current != 0:
        base_n = map_to_base[current % len(base)] + base_n
        current = current / len(base)

    return base_n

if __name__ == '__main__':
    sys.exit(main())

And giving this input (the first line is number of the following lines):

4
9 0123456789 oF8
Foo oF8 0123456789
13 0123456789abcdef 01
CODE O!CDE? A?JM!.

I have the correct output:

Case #1: Foo
Case #2: 9
Case #3: 10011
Case #4: JAM!

Of course this is one of the practice problems and you should try to solve it by your own (otherwise it’s useless to try to join the event).

And I’ve been 6th. So I won a shared 2 hosting plan at webfaction and a 12 pack of G33K B33R caffeinated root beer (still trying to understand what this is exactly, anyway) from bawls. Anyway, here follows a short resume of what happened from Saturday through Tuesday (if you’re asking yourself why it didn’t ended on Sunday, well, keep reading).

The competition began very well, I worked normally for the first part of the day but then I had to stop for a while. When I came back, svn and djangodash website was not working anymore. I initially thought that it was some connection issue but when I saw that other sites were working properly so they definitely had some problems.

I just waited, then gone sleeping. In the morning I received in my mailbox a message that informed me of a big power outage in The Planet datacenter where webfaction hosts a lot of their server (among the which there was the djangodash one) caused by power generator’s explosion. Then the competition has been delayed for other two days, so I decided to take a breath and wait ‘till the svn would came back. But that didn’t happen on Sunday, so after a while I chosen (as the mail suggested) to work locally without committing anything at least until the svn return.

Then Monday came and I had other things to do, so I had to postpone djangodash for the evening when I’d freed myself from other, most urgent things. On Monday I did a very little coding, as well on Tuesday. So at the end of competition I cannot complete my project, and not even reach the 50% milestone.

Today I discovered that I was one of the winners (ok not really, 6th place was not really a good place, but at least I tried) and I really have to thanks the organizers for this event and hope to join another djangodash next year. Maybe, as I said to one of them in an email thread, hosting the site/svn in two different datacenters, just to be insured against eventual thunderstorms, tornado, earthquakes and so on…). I have to say that I really enjoyed the whole thing, and hope to have more competitors next year!

If you want to get more news about final process of djangodash with some stats, read this article on the Toast Driven website (that’s the company that ran the dash).

As probably many of you already knows, on May 31 will begin the Django dash competition. Djangodash is:

[…] is a chance for Django enthusiasts to flex their coding skills a little and put a fine point on “perfectionists with deadlines” by giving you a REAL deadline. 48 hours from start to stop to produce the best app you can and have a little fun in the process.

I’ll be participating, so if you haven’t registered yet, do it now! And don’t forget to check out the cool prizes :)