There are few variations of the same problem but most common one is named “0-1”: you can’t split the weight over two or more carriers or bags but either you take the whole weight or you leave the object where it is. Mathematically talking, consider a set of objects, each item is worth and weights with . Then the goal is to maximize the following function:

But keeping the following constraint (being the maximum weight we can carry):

The first problem we have to face then is how we should generate the items and how the value of a solution should be calculated. The following example, as the other that will follow, it’s written in Python but it’s quite easy to understand so porting to another language wouldn’t be that hard. I chosen to generate 50 objects with values that range from 1 to 99$ (both ends included) using a uniform distribution (if you don’t know much about statistic, it means that all the values are equally distributed among the objects). The same with the weights except they range from 1 to 20 (the choice of the weight’s unit measure is left to you).

def generate_items(n_items=100):
    "Generate a list of items that could be stealed"
    items = []
    for n in range(0, n_items):
        # use a uniform distribution both for values and for weights
        cost = random.randint(1, 100)
        weight = random.uniform(1, 20)
        items.append((cost, weight))
    return items

So the items are nothing other than a list of pairs in the format for every object . Probably a better and more realistic dataset would have used a normal distribution for the weights but it’s trivial to change the generation to function to work in that way. For our purposes the uniform distribution does its job.

As we said above the problem is NP-complete so we usually need to visit the whole search space to get the optimal solution which can be quite big when as the number of objects grows. Here comes the simulated annealing. We won’t visit the search space extensively, but we’d rather generate solutions. Indeed, it is a stochastic heuristic search algorithm. An heuristic is a function that measures how much something is good or bad, and stochastic means that we move more or less in a random way into the search space. In practical terms it’s not greedy as that it doesn’t always follow what the heuristic says but rather randomly search where the heuristic function points to. For example consider the needle in the haystack situation: an exaustive search method would take every straw piece, check that it’s not a needle, put it apart and repeat those moves until you don’t find the needle. You don’t want to proceed in this way, you’re more likely to end in less time if you look randomly in the haystack and if somethings stings you while you’re holding straw then search into that straw piece, because there may be the needle in there. In the knapsack problem we do have the heuristic, and it’s the function above that, for each solution (admissible or not), says how much it’s worth. In this case we define an admissible solution as one that it’s not too heavy.

An useful (and classic as well) example is the one of the blind hill climbing. You’re blind and stuck on a hill and you need to reach the top. A good principle that could lead you to the top is to touch the terrain and always follow the rising path. It could, because if you’re on a rock than you surely haven’t reached the top but the principle above doesn’t apply: you reached a local maximum (invert the things and you get the same thing for a local minimum). Simulated annealing avoids these problems by trying worsening moves from time to time: even if this may not sound like a good move it helps avoiding the problems we described above.

In the function above once in the middle we could choose to take the left maximum (which is a local maximum). Using hill climbing we’d be stuck on that because we wouldn’t try other paths.

Simulated annealing takes its name from the same process that metals go through when cooling from a melting point. Indeed, the cooling process consists of several particles that changes energy states (this statement may not be accurate or be inexact at all, but please forgive me as I never studied those things and I all know in this field comes from simulated annealing algorithm itself), in particular we can calculate the transiction probability from one state to another. Considering two energy states and and a temperature , switching from to has probability:

Where is a constant called Boltzmann’s constant. But then, how do we apply those statements to our problem (or a combinatorial search problem, in general)? The most important concept to grasp is the energy switching one. As a particle change state, a solution might change. Indeed for the knapsack problem there are many admissible solution, each one with an associated earning. Of course we’d prefer the one with the higher earnings (and simulated annealing will help us find that) but still it’s perfectly acceptable to go from a solution to another as long as the other solution continues to be admissible.

So here it is what the simulated annealing does: if you find a better item go on an take that path (under this circumnstance, behaves just like the hill climbing), otherwise change state with probability (you may notice that the Boltzmann’s constant is missing, indeed that constant applies mostly to thermodynamic when dealing with different metals). The effect that the temperature scaling has is that at higher temperatures it’ll try worsening moves quite often while on lower temperatures the probability to do worsening moves is lower so when temperature tends towards 0 it behaves quite like the hill climbing algorithm.

def simulated_annealing(solution, items, max_weight):
    "Apply the simulated annealing for solving the knapsack problem"
    best = solution
    best_value = compute_cost(solution, items)[0]
    current_sol = solution
    temperature = 1.0

    while True:
        current_value = compute_cost(best, items)[0]

        for i in range(0, COOLING_STEPS):
            moves = generate_moves(current_sol, items, max_weight)
            idx = random.randint(0, len(moves) - 1)
            random_move = moves[idx]

            delta = compute_cost(random_move, items)[0] - compute_cost(best, items)[0]

            if delta > 0:
                best = random_move
                best_value = compute_cost(best, items)[0]
                current_sol = random_move
            else:
                if math.exp(delta / float(temperature)) > random.random():
                    current_sol = random_move

        temperature = TEMP_ALPHA * temperature
        if current_value >= best_value or temperature <= 0:
            break

And finally, that is the simulated annealing. You start from a temperature of 1.0 then you have a certain number of cooling steps, in every one of them you extract a random item from the neighbours and, if the item is better than the current best item then it becomes the new best item (and the new local solution). If the new item’s value is worst than the current best then update the current local solution with the probability expressed above. After the cooling steps the temperature is decreased with an exponential decay (usually it is ).

In the example above you don’t wait for the temperature to be 0 but you leave the loop if after all the cooling steps there hasn’t been any improvement. How big must be the number of cooling steps and the value it’s a fine tuning problem. A different approach that allow to get rid of the cooling steps is to make the temperature get cold slowly ( and is a very small value like 0.01). Besides, a common improvement is to cache the moves within the cooling steps unless you found a new best value or changed state.

You can see that the most critical points are the neighbour generation and the cost computation. While the neighbour generation could be cached like I said above, the cost computation could be replaced with a probability estimate in order to reduce the time per cooling step.

But how do you apply the algorithm? If an empty solution is acceptable then you can just start with that and let the neighbour’s generator to create a solution, but usually you start with a greedy solution (found through the hill climbing, for example) or from a random one.

Here follows the complete code. I used 1000 cooling steps and a value of 0.8. I start from a random solution whose behaviour is not that bad considering how much time it takes to compute the solution. Indeed often random algorithms perform really well on combinatorial algorithms, see stochastic search for some more informations.

#!/usr/bin/env python
import math
import operator
import pprint
import random
import sys

COOLING_STEPS = 1000
TEMP_ALPHA = 0.8

random.seed()

def generate_items(n_items=100):
    "Generate a list of items that could be stealed"
    items = []
    for n in range(0, n_items):
        # use a uniform distribution both for values and for weights
        cost = random.randint(1, 100)
        weight = random.uniform(1, 20)
        items.append((cost, weight))
    return items

def main(args):
    items = generate_items()
    pprint.pprint(items)

    start_sol = generate_random_solution(items, max_weight=40)
    print "Random solution: %s" % start_sol
    print "value: (cost: %d, weight: %f)" % compute_cost(start_sol, items)

    solution = simulated_annealing(start_sol, items, max_weight=40)
    print "Final solution: %s" % solution
    print "value: (cost: %d, weight: %f)" % compute_cost(solution, items)

    return False

def generate_random_solution(items, max_weight):
    "Generate a starting random solution"

    # generate a random solution by adding a random item
    # until we don't get over the weight
    solution = []
    while compute_cost(solution, items)[1] <= max_weight:
        idx = random.randint(0, len(items) - 1)
        # skip duplicates
        if idx not in solution:
            solution.append(idx)
    # last item makes us get over the weight so simply remove it
    # we'll look for better results after
    solution = solution[:-1]
    return solution

def simulated_annealing(solution, items, max_weight):
    "Apply the simulated annealing for solving the knapsack problem"
    best = solution
    best_value = compute_cost(solution, items)[0]
    current_sol = solution
    temperature = 1.0

    while True:
        current_value = compute_cost(best, items)[0]

        for i in range(0, COOLING_STEPS):
            moves = generate_moves(current_sol, items, max_weight)
            idx = random.randint(0, len(moves) - 1)
            random_move = moves[idx]

            delta = compute_cost(random_move, items)[0] - \
                    compute_cost(best, items)[0]

            if delta > 0:
                best = random_move
                best_value = compute_cost(best, items)[0]
                current_sol = random_move
            else:
                if math.exp(delta / float(temperature)) > random.random():
                    current_sol = random_move

        temperature = TEMP_ALPHA * temperature
        if current_value >= best_value or temperature <= 0:
            break
    return best

def generate_moves(solution, items, max_weight):
    """
    Generate all the ammissible moves starting from the input
    solution
    """
    moves = []
    # try to add another item and save as a possible move
    for idx, item in enumerate(items):
        if idx not in solution:
            move = solution[::]
            move.append(idx)

            if compute_cost(move, items)[1] <= max_weight:
                moves.append(move)

    # try to remove one item
    for idx, item in enumerate(solution):
        move = solution[::]
        del move[idx]
        if move not in moves:
            moves.append(move)

    return moves

def compute_cost(solution, items):
    """
    Return a tuple in the format (id_item1, id_item2, ...)
    for the input solution
    """
    cost, weight = 0, 0
    for item in solution:
        cost += items[item][0]
        weight += items[item][1]
    return (cost, weight)

if __name__ == '__main__':
    sys.exit(main(sys.argv))

The results are suprising. For three different set of 100 items, each one with its own value and weight, here are the results:

$ python sa.py
Random solution: [98, 71, 95]
value: (cost: 44, weight: 27.001685)
Final solution: [71, 95, 67, 9, 41, 33, 27]
value: (cost: 229, weight: 39.791386)

$ python sa.py
Random solution: [38, 16, 62, 31]
value: (cost: 124, weight: 36.863846)
Final solution: [38, 16, 62, 31, 5]
value: (cost: 194, weight: 38.970745)

Random solution: [61, 44, 48, 38]
value: (cost: 293, weight: 30.357135)
Final solution: [61, 44, 48, 38, 37, 5, 2]
value: (cost: 421, weight: 39.331549)

We usually don’t leave much free weight and the quality of the solutions found is quite good considering that the time required to compute that is ~0.5s.

They’ve been loyal: when you signed up the registration form, they asked you if you were comfortable with APIs changing or an instable system. That’s what you’ll find once you get your sandbox account. APIs are there but haven’t been fully documented yet and most of your work when developing some robot/gadget will be in exploring the API sources (they’re open source, yau!) or searching for some examples on the samples gallery, which is an invaluable resource by the way.

Debugging is hard too, given that you can’t test what you done locally but you have to upload your code to AppEngine to see if it works (actually AppEngine is the only platform they accept requests from, but they plan to allow every host that talks the wave protocol in the future). This means that if, for example, there’s some typo in the code (i.e.: appendText() rather than AppendText()), you’d know only by looking at the AppEngine logs.

Be prepared to experience casual failures too. Sometimes your robot is working correctly and is receiving the whole wavelet (which is the whole conversation thread), but its response is ignored by the server for some unknown cause.

Anyway, even though there’s still some clear work in progress, I felt like the whole thing was quite exciting both from the user and as the developer point of view. The event model they thought for the external applications perfectly fits the nature of The Wave and gives room for some nice asynchronous applications. Hopefully, we’ll meet on Google Wave soon.

First thing, no matter how good your data structure is or how well you implemented that, sooner or later you’ll meet a speed barrier that even the best data structure for the job can’t beat. That doesn’t mean you don’t have to think about what data structure to use, besides I’m really passionate about finding the right data structure for the job so if I have the chance to deal with this kind of problems seems like I won the lottery. Sometimes the algorithm you’re using simply has some limits that can’t be beaten unless you radically adjust your algorithm or you completely change it with some a better-algorithm-for-the-job. Say we have an implicit, non-weighted graph, but we know that at most we’d be expanding fifty nodes of about a hundred bytes each and the expansion of every node comes cheap as computational cost. Now you want to find a particular node within that graph and along with it you want to find the shortest path to the starting node. Given these considerations, what algorithm would you use in this case? I’d go for breadth-first search, since we don’t lose much time expanding the nodes (computational cost of each node expansion), we have relatively few nodes that, unless we’re running on some special hardware that is somewhat limited (even though today even those automatized toilet’s chips can hold 5Kb in memory), will take a very little amount of memory.

Now suppose we still have that implicit graph, with our fifty nodes of about a hundred bytes each. But now we know that generating every node is very expensive in terms of computational costs. The BFS above could still be applicable under certain circumstances but we’d better be looking for alternatives. You can hold the graph in whatever data structure you want but unless you decide to change the algorithm with something better there’s nothing you can do about. Of course you can gain some speed by improving your data structure but that’s not the point since what makes the shortest path search slow is the generation of a new node. So what you have to do? Try to generate as few nodes as possible. Indeed, say our nearest solution is ten edges far from the starting node, with BFS you’d expand first all the nodes which have a direct connection to the starting node (so one edge), than from each of these nodes you’d be expanding other nodes that now will have two edges separating them from the starting nodes, and so on until you expand all the levels and get to the solution which is separated from the starting node by ten edges. You had the solution eventually, but it came at an expensive price. What are the alternatives then? Say we know, for each node we expand, a numerical value which says how good that node is in terms of distance from what we’re looking for. In this way we can follow only the good leads and leaving the bad paths out of our research (that, though, is not entirely true, and I’ll say why in a minute) and still, we have limited the number of expanded nodes and as result we had a great speed up. There’s an entire category of algorithms that are based upon the principle that you know something of your problem that can help you out in some cases and these are called informed algorithms. One of those who comes to my mind is A* which is quite simple to implement too. These algorithms are based upon the assumption that you can make some estimation of how good the current state is. Indeed, most of these algorithm’s accuracy comes from how good the function that gives you that estimation is. This function is called the heuristic function. But the heuristic function is an estimation and it’s likely to be wrong in some cases. So you still end up expanding more nodes than necessary and following some wrong leads, but now you save yourself from exploring all the wrong leads that the previous BFS would have forced you to do.

For the project I was talking above, at a certain point we hit the limit. We tried switching data structures initially and that saved some time but still we weren’t able to come up with any consistent time reduction. Until we switched algorithm, then we gained something like a 80% speed up and we even had the chance to use some simpler data structure which allowed us to exploit some stuff like caching which I suppose is a large part of that 80%.

Saying that, here comes the second lesson: know your environment. You have to know how your processor works, what’s the difference between, say, a L2 and a L1 cache, why disk access is slow and how it works, how the paging works and so on. These things can make great room for optimization if you know how to use them. Caching, for example, is probably one a thing that may give a great help to you. If you know what the cache line is and how big is on your processor, you’ll know how and when you can exploit the data locality principle. Knowing that disk reading is not that slow as it’s said could really help you (I heard you crying: I did not mistyped, disk read ain’t slow, what’s makes this operation slow is the time the disk’s head takes to position itself. Once we got the head in position, reading a whole block is probably faster than you imagine. Of course, I’m talking about old-style mechanical disks, with these shiny new solid state disks is another story). In the same way it can help knowing that your program will not cause many page faults or won’t fragment the allocation memory because your data structure doesn’t fit in one page or your algorithm makes thousand of allocations/deallocations of different size. This is very important, even though I got to admit is one of the topics I’m lagging behind.

Then you chosen the best algorithm in the world, you resurrected the dead in order to fit your data structure in exactly one page and you used condoms while you were coding, but still your program is very slow and you still can’t find the reason. Well I tell you, probably you’re using the wrong data structure but at this point I guess you’d know. With the project I said above, we experienced exactly this. We did everything we can to speed up the algorithms, we optimized everything could have been optimized but yet our program was slow until we realized that something was a real bottleneck for our purpose. Big Oh analysis it’s really useful, but you got to take into account that that little ‘n’ in that big ‘o’ is meant to be big. Indeed, that gives you an idea of how the data structure works as n grows, so that two same algorithms or data structures which both have a O(nlogn) complexity can be very different. In our case, the bottleneck was the STL map implementation. It turns out that, under the hood, it is a red-black tree that was everything but fast in our program. We were spending 30% of the time within the STL map looking for the key. The problem here really was that we hadn’t many values within the map, most I’ve seen were ten items but we had an average of three or four items. When we switched from the usual map to a very simple hash, we had a terrible speed improvement.

Next thing is optimizing little stuff. This won’t yield greet speed ups but yet, they’re useful. We gained about one second by only switching one function’s calling convention to fastcall and we had even greater benefits by forcing the inlining of some other functions.

Other than that, having to work on a large C++ code base has been challenging, most because of some C++ gotchas (i.e.: why something like string fn() {} returns NULL implicitly?) even though I come from a assembler/C background and I have a solid OO theory base behind. In the end, though, it has been probably the funniest problem I faced in the latest years and will be eventually be open sourced after the summer.

On first day, there were two keynotes: “A retrospective of how the community helped build Python 3.0”, held by Guido van Rossum and “Zen and the art of Abstractions’ maintenance” by Alex Martelli. I can just say that they were two extremely interesting talks which by the way weren’t diving too much — or any at all as in Guido’s talk — into code.

On the second day I really enjoyed two talks: “Erlang + Python, joining two worlds” by Lawrence Oluyede and a really great talk by Raymond Hettinger, “Easy AI with Python.” The former left me with a great curiosity about the functional languages world, while the latter really impressed me with how easy is to solve certain AI problems with Python (I solved many of the problems Raymond talked about previously, but never in Python and never really thought about even trying to).

On third day the Antonio Cangiano’s talk was enlightening. Even though it wasn’t really Python specific, he has given a great insight of how you can, well, “become rich with Python.”

Unfortunately I didn’t follow the Sunday afternoon’s talks since my airplane was leaving at 3.00pm, but at the end I can say that this was an incredible experience that I hope I can make again next year. And as a side note: the food was marvelous.

In the meantime, I’ve been doing several experiments on this topic mostly about the memory usage and the performances of such system on limited hardware. This practically means implementing the algorithms you’ll be using and measuring the computational time they require.

One of the most common thing that our information retrieval engine have to do is to take a document and compress it, but considering that:

• this is a fundamental piece of this IR engine
• it will be used very often
• it’s not rare to process very large documents

You’ll get that this operation should be as efficient as possible.

I chosen to go down with zlib as my compression library for mainly two reasons:

• it’s already included in Python (this is not really a strong point since better compression algorithms are included in Python too)
• offers the best compromise in speed/compression ratio

Given the above considerations, let start coding our compression system.

We will use as our document example the PDF specifications, available at the Adobe Development Center (this is the file) that are 8.6Mb large.

So let start doing the things the basic way:

#!/usr/bin/env python
# compress1.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    out = ''
    for line in input_fd:
        out += cobj.compress(line)
    out += cobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    out = ''
    for line in input_fd:
        out += dobj.decompress(line)
    out += dobj.flush()

    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

By running this program and performing a very basic profiling we get some indications:

kratorius@becks:~/compress$ time ./compress1.py compress PDF32000_2008.pdf compr.zlib
real    0m2.517s
user    0m1.496s
sys     0m0.060s

kratorius@becks:~/compress$ time ./compress1.py decompress compr.zlib decompr.pdf
real    0m0.640s
user    0m0.537s
sys     0m0.085s

We need 2.5 secs in order to compress a file smaller than 10Mb. This is quite unacceptable, since it means that we’re processing about 3.5Mb per second; so we need to understand what we’re doing wrong. I can spot at least two big errors in this script:

1. we’re reading the input file line by line that isn’t very efficient since in this way we’re accessing the disk multiple times (not counting that we are also processing the compression stuff line by line, that it’s not efficient and hasn’t so much sense in a binary file like our PDF)
2. we keep our compressed object in memory until we finish the compression, and this means that if the script would run faster, we’d still have a very high memory usage that is not optimal

So here it is the new version of our compression script that address the issues above:

#!/usr/bin/env python
# compress2.py
import zlib

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.compress(input_fd.read(), compression_level)
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    out = zlib.decompress(input_fd.read())
    output_fd.write(out)

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

Let perform our basic profiling again:

kratorius@becks:~/compress$ time ./compress2.py compress PDF32000_2008.pdf compr.zlib
real    0m1.668s
user    0m1.337s
sys     0m0.079s

kratorius@becks:~/compress$ time ./compress2.py decompress compr.zlib decompr.pdf
real    0m0.561s
user    0m0.394s
sys     0m0.086s

We are now reading the whole input file in memory (minimizing the disk accesses), compressing everything in memory and writing the compressed file to the output in a single shot. We got a high speedup in this way but we have just increased our memory usage since now we’re keeping in memory both the input and the compressed file. This could be optimal if we’re processing small files, but since we need to have a generalized approach, this solution is not that good.

We can do better. And we’ll do better in the third try:

#!/usr/bin/env python
# compress2.py
import zlib

READ_BYTES = 2097152 # 2Mb

def compress(input_path, output_path, compression_level=6):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    cobj = zlib.compressobj(compression_level)
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(cobj.compress(rd))

    output_fd.write(cobj.flush())

    input_fd.close()
    output_fd.close()

def decompress(input_path, output_path):
    input_fd = open(input_path, 'rb')
    output_fd = open(output_path, 'wb')

    dobj = zlib.decompressobj()
    done = False
    while not done:
        rd = input_fd.read(READ_BYTES)
        done = rd == ''

        output_fd.write(dobj.decompress(rd))

    output_fd.write(dobj.flush())

    input_fd.close()
    output_fd.close()

if __name__ == '__main__':
    import sys
    args = sys.argv[1:]

    options = { 'compress': compress,
                'decompress': decompress,
    }

    input_path, output_path = args[1], args[2]

    try:
        options[args[0]](input_path, output_path)
    except (KeyError, IndexError):
        print("Invalid arguments")

And we finally reached our goal:

kratorius@becks:~/compress$ time ./compress3.py compress PDF32000_2008.pdf compr.zlib
real    0m1.325s
user    0m1.226s
sys     0m0.070s

kratorius@becks:~/compress$ time ./compress3.py decompress compr.zlib decompr.pdf
real    0m0.534s
user    0m0.404s
sys     0m0.119s

This last try works because we’re still minimizing the disk accesses for small files (we’re reading 2Mb chunks per time) and this time we’re reducing the memory usage since:

• we read a 2Mb block from our input file
• we compress the read input
• we write it directly to our output file

I’m sure there’s still room for improvement but at this point we can be quite happy of our achievement. You can find the final script that performs error checking and file locking here (file locking works only on UNIX systems though, on Windows you should just comment the fcntl lines out). As always, suggestions are welcome.

In this week I learned a lot of things about information retrieval, text categorization, natural language processing and machine learning. But the most relevant thing is: the principles are easy, their implementation is not. The fact is that most of the techniques are relatively simple but you usually have to deal with very large datasets and this could be challenging, since one of the main requirements about information retrieval is time. It’s really much more important that you give less results in one second rather than giving better results in one hour. No one will ever care to use your system if it takes an hour to get some result. And if you’re considering to store your data in a database forget about normalization, it wouldn’t really take you anywhere.

Talking about storing informations, you know that if you’re dealing with documents most of the words are the so called stop words. Those stop words are words that doesn’t really mean anything, but they help the readers to get a better text flux. Classic examples of stop words are articles like “the”, “a”, “an” or logic connectors like “or” and “and”. These words are so common that their presence is quite useless since they’re are… everywhere. If you’re going to study information retrieval than you’ll learn about a weighting technique called tf-idf that gives a weight near to 0 to these words, but since you’d probably use a reverse index for words (an index that given a word, tells you in which documents that word appears) you can understand that this would take a lot of space if you’re going to include stop words.

So one of the biggest issues until now is that you’re going to deal with extremely large datasets, so you have to strip as many things as possible. Now consider those words: “fishing”, “fishes”, “fish”. They all talk about “fish”, and an user that is searching for “fish” would probably be interested in “fishes” or “fishing” as well. Additionally, it’s useless to store three words that are almost identical. So here comes the stemming that, by quoting the related wikipedia page, is the process for reducing inflected (or sometimes derived) words to their stem, base or root form – generally a written word form. Fortunately, if you’re dealing with english texts, there’s the Porter algorithm that is the state-of-the-art algorithm for this sort of things. But that works only with english, so if your documents are written in another language or they are written in multiple languages, things are going to be complicated.

This leads to think about the problem of the language identification. How do you know if some text is written in a language or in another just by looking at it? Of course you can describe the document’s language with some kind of meta tagging, but not all the documents have this kind of description, just think about the web. There are some kind of statistical methods based upon the classification of n-grams but I haven’t deeply investigated about them yet, so I can’t really say anything.

Now you got your collection of documents that match a certain query. Now: how do you know what document is more relevant than another (in other words: how do you rank pages)? You got two alternatives (well, probably more, but I know just these at this moment): the tf-idf that we said above and the cosine similarity. The latter is an interesting one: consider the tf-idf vectors of the documents, then consider the query as a document too. Now plot those tf-idf vectors and measure their cosine of the angle between them. The more you’re near to 1, the more relevant is the document.

There are a lot of other important things that need to be said like the precision and recall concept, but that’s enough for now. I’ll talk about this another time.

Anyway I’m doing an experimental project named django searchable. It’s a pluggable app for django that implements an information retrieval engine based on tf-idf weighting. Play with it if you’re brave enough.

It has a very basic start/stop/restart interface like normal startup scripts. Let see an example of basic usage:

kratorius@becks:~/prj$ sudo sh startserver.sh start
Starting fcgi process... done!
kratorius@becks:~/prj$ sudo sh startserver.sh stop
kratorius@becks:~/prj$ sudo sh startserver.sh restart
fcgi process is not running
kratorius@becks:~/prj$ sudo sh startserver.sh start
Starting fcgi process... done!
kratorius@becks:~/prj$ sudo sh startserver.sh restart
Starting fcgi process... done!

And here it is:

#!/bin/bash
# Start and stop a django fcgi process
# Giuliani Vito Ivan <giuliani.v@gmail.com>

# project directory
PROJDIR=`pwd`

# user owner (usually www-data)
USER_OWNER="www-data"

# group owner (usually www-data)
GROUP_OWNER="www-data"

# extra python path (leave empty if unneeded)
PYTHONPATH="../python:.."

# do not edit anything below
PIDFILE="$PROJDIR/technosec.pid"
SOCKET="$PROJDIR/technosec.sock"

start_fcgi()
{
	if [ -f $PIDFILE ] || [ -f $SOCKET ]; then
		echo "The fcgi process is already running, please stop"
		echo "that before running another process"
	else
		echo -n "Starting fcgi process... "

		nohup /usr/bin/env - \
			PYTHONPATH=$PYTHONPATH \
			python manage.py runfcgi socket=$SOCKET pidfile=$PIDFILE > /dev/null 2>&1 &
		if [ $? -eq 0 ]; then
			sleep 1
			chown $USER_OWNER:$GROUP_OWNER $SOCKET
			chown $USER_OWNER:$GROUP_OWNER $PIDFILE
			echo "done!"
		else
			echo "failed!"
		fi

		return $?
	fi

	return 1
}

stop_fcgi()
{
	cd $PROJDIR
	if [ -f $PIDFILE ]; then
		kill `cat -- $PIDFILE`
		rm -f -- $PIDFILE
		rm -f -- $SOCKET

		return 0
	else
		echo "fcgi process is not running"
		return 1
	fi
}

restart_fcgi()
{
	stop_fcgi
	if [ $? -eq 0 ]; then
		start_fcgi
	fi
}

case "$1" in
	start)
		start_fcgi
		;;
	stop)
		stop_fcgi
		;;
	restart)
		restart_fcgi
		;;
	*)
		echo "Usage: $0 {start|stop|restart}"
		exit 1
		;;
esac

exit 0

Just in case you may want to download it, here it is.

I did this for generating a django quick reference (here it is) since it has a lot of functions and I know what’s their purpose, but I can never remember the names (and now two A4 papers are right in front of me).

If you’re interested in this, check out the google code project page and grab your copy from the SVN repository.

There are tons of things that can be changed/optimized (i.e.: add some optional short explanation of the function, add more examples, easier way to change colors, …) but now the code is working quite well so that can be already useful to the people out there.

What I wanted to achieve was basically this:

I thought at various techniques on how to realize that: fixed image, text with alternated background, experiments with PNG transparency, ecc…. All of these things have their pros and cons, but in the end I came up with a simple solution that required just few lines of CSS code and had few cons (in respect to the others).

Let see how I did it. First of all, we need a container for our text. I used a div on the which I set the text id. In #text we’re going to put our text. We’ll use absolute positioning within the div, so #text has to have position: relative set.

Next, we need two spans with the same text within; on these we’re going to set the absolute positioning I said before, setting top: 0; left: 0; on them. And now here is how the trick works. By having absolute positioned the two spans and having put them at the same coordinates (with the same font size), they’re going to overlap each other. Thanks to absolute positioning we can set the height on the spans, and this is the core of the trick, since to make it work we’ll have to set the first span’s height to half the height of the second span.

But that’s not all. If we stop here nothing will work. We also have to use overflow: hidden and z-index: 1000 on the first span. Just as side note, 1000 is not really a mandatory value for z-index, you can use even 1 but 1000 is pretty standard if you want that the class is always on the top.

In this way this is the final result. I really like it but cannot implement on this design as doesn’t integrates well the whole layout. Anyway, it has some cons: you have to manually tweak the font-size and height’s values as it’s not possible to use relative sizes and your code is going to have the text repeated twice, though this can be easily fixed with a bit of javascript.

I tested this under Firefox 3.0.2, Explorer 7, Opera 9.25 and Safari 3.1.2 and it works without any hack though I’m quite sure that it’s going to have some issues with older versions of Explorer.