nsyll just released the drupal urlBorg module!

TwitVim, a Vim plugin that allows you to post to Twitter and view Twitter timelines, has now support for urlBorg. Cool! Details at mortonfox.livejournal.com.

Nektarios is developing a drupal plugin that will make use of urlBorg. He asked me why, urlBorg creates a new short URL every time he makes an API call, even if the target URL is the same. Wouldn’t it be reasonable to get the same short URL? Isn’t it a waste of resources (i.e. storage space) to create and store a new short URL every time?

This is one of the main design decisions I had to do when I started developnig urlBorg. And I’ve changed my mind a couple of times. But I think that creating a new short URL is the right way to go. Here is why.

The creation of a short URL is a “gesture of attention” (if you are not familiar with the term, read Geting a piece of the action: The attention economy). An if a URL shortener is worth something, this is because it is a good database of two kind of “gestures”:
- the one is when someone manifests his interest on a specific URL, by creating a short URL for it.
- the other is when someone clicks on the short URL.

urlBorg keeps track of both. The click stats are visible to anyone, the just have to add /i at the end of a urlBorg short URL, like this Comments (1)

summize.com is a search engine for twitter -and a great one!

urlBorg now queries summize.com for each short URL and displays the results in the preview page. This way it is easier to guess or understand what the link is about, before going to the actual page.

urlBorg lets users see more detailed stats now.

There are many ways to access the click stats:

1. Click the “more info” link at the bottom of any “preview page”
2. Login to your account and click the short URL on any of the “created” or “clicked” links.
3. append /i at the end of a short url. ex ub0.cc/6/0h/i

The graphs are created using the Google Chart API and IMHO the look quite nice

I asked urlBorg users to rate the app in the AppEngine application gallery and got some enthusiastic comments:

One of the best url shortening tools - jim.hellas - May 24, 2008
UrlBorg has a very simple interface, provides personal account and helps you keep track of your urls. Simply the best

The best out there! - kostis_at_quatzacoalt.com - May 24, 2008
My choice for URL shortening! Useful features! I’m sure more are on the way!

Great stuff - stefanos.kofopoulos - May 24, 2008
…now running on scalable environment.

urlBorg rules! - niels.vaneck - May 24, 2008
Scalable url shortening in combination with user accounts makes for a very useful service.

Wow! Thank you!

urlBorg has two new features:

1. It will keep a record of all the “short URLs” you’ve created, with stats (number of clicks). This is a nice way to keep track of the URLs you’ve created but also to see how many clicks each one of them had.

2. It will keep a record of all urlBorg short URLs you’ve clicked on. This is an optional feature, you have to enable it explicitly. I often find myself looking for “that link to a funny video someone posted on twitter yesterday”. Well if this someone used urlBorg to create short links, now I will find it in my history.

To use both of them, you have to login of course. But I took advantage of AppEngine’s Users API which means that if you already have a GMail account you login using it -no registration, etc. (I know I should have used OpenID, but for now it was so much simpler to use Google’s authentication -but OpenID is on the ToDo list).

Now, if someone could advice me on what is the right format to export the attention data gathered, so that users are able to reuse them and move them around, please let me know.

At the top of each page on urlBorg you will se a link named “account”. Click on it and login using your GMail account.

You will be redirected to your urlBorg account page, where you will se the option “enable previews”. Previews are simple pages that look like this (here, I take advantage of a hidden urlBorg feature: if you add /p at the end of a short URL you “force” the preview to show up). Their obvious value is to protect you from going to pages you wouldn’t like to visit, but their URL is hidden by urlBorg.

I think “previews” can be much more. Here are a couple of features I already implemented.

MP3 files
If a URL goes to an MP3 file (determined by the MIME type reported by the hosting server), the preview will also show you flash mp3 player that allow you to listen to it. Here is an example.

maps
For this one, I have to thank Raphael from U.[lik]. Raphael is has been using urlBorg to share links to Google Maps (those links can be long and ugly). Now, if you link to a page in Google Maps or Yahoo! Maps, the preview will also display a link to the other service, where you will se the same map.

Here is a Google Maps expample (again, note the /p at the end of the URL) and a Yahoo! Maps example. You don’t have to do something special. urlBorg will examine the structure of the link and display automagicaly the “extra info”.

So, what do you think?

So, urlBorg has been rewritten in Python and is now hosted on Google AppEngine (make a note, the new address is urlborg.com).

But why build “yet another URL shortening service”, when it’s so easy to build one? Any web developer could build one in less than an hour, couldn’t they?

The truth is that building a URL shortening service is a trivial task. Building one that could scale is not. I designed urlBorg having in mind “will it work if it made it to TechCrunch or if CNN.com made extensive use of it?” Building such a service is not trivial, believe me. (And I won’t know if urlBorg will make it either, but I think it will.)

But scale wasn’t my only motivation. I believe there’s a lot of space to add value to such a simple service. A quick look at the API will reveal some of my ideas -urlBorg goes beyond returning a short URL.

More details to come soon

As I’ve mentioned before, I’m trying to migrate urlBorg to Google AppEngine. urlBorg needs to count many things, like clicks on a short URL, etc, so I really need a scalable and efficient way to implement counters. This is not as trivial as it sounds in the Google AppEngine environment.

This post is actually the result of a good discussion done here

Here is the code I’ve come up with.
An example usage would be as simple as adding a line like (where page_id is a unique string identifying each page)

Acc(page_id).acc()

in each one of your pages. Getting the total coun is as simple as

Acc(page_id).val()

(Due to the way the total count is calculated, this may not give accurate results if you are in the middle of a traffic spike, but it’s good enough for web analytics usage)

class AccVals(db.Model):
       cluster = db.StringProperty(required=True)
       count = db.IntegerProperty(required=True)
       updated = db.DateTimeProperty(auto_now=True)
       rand = db.FloatProperty()

class Acc():
       def __init__(self, name,init=0):
               self.__sec = 0.1
               self.__name = name
               self.__init = init

       def inc(self):
               def trans(key):
                       obj = AccVals.get(key)
                       obj.count += 1
                       obj.put()
                       self.__val = obj.count

               q = db.Query(AccVals).filter('cluster =',self.__name).filter('rand >', random.random()).get()
               if (q):
                       if (datetime.datetime.now() - q.updated < datetime .timedelta(0,self.__sec)):
                               obj = AccVals(cluster=self.__name,count=self.__init, rand=random.random() )
                               key = obj.put()
                       else:
                               key = q.key()
               else:
                       obj = AccVals(cluster=self.__name,count=self.__init, rand=1.0 )
                       key = obj.put()

               db.run_in_transaction(trans,key)
               return self.__val

       def val(self):
               total = 0
               q = AccVals.all()
               q.filter('cluster =',self.__name)
               for r in q:
                       total += r.count
               return total

It behaves relatively good and looks like it can scale no matter how
much traffic or traffic spikes you have.

If you look into it, you will see that a “counter instance” is chosen
in random. You may be tempted to use the “instance” that was updated
longer in the past ( order(’-updated’).get() ), but it turns out that
when you have a traffic spike (or whatever it is your counters count)
the indexes are not updated soon enough and this will return the last
records that were updated It looks like selecting a random
instance is no big deal in low traffic and works much better in high
traffic. I’ve also seen that after a while, you end up with the number
of counter instances that are required to handle the traffic of the
specific counter with few transaction collisions.

There is one interesting point: the value of self.__sec. I set it to
0.1 seconds, but this is just a value that looked good after some
tests. I have the impression that this value is *related* to some kind
of “global AppEngine constant”, measuring the time it takes for a
transaction to complete and safely propagate to the rest of the
infrastructure. I guess this varies, depending on the resource
allocation done for a specific app. Could someone from the AppEngine
development team give us some insight on this?

As I’ve mentioned before, I’m a Python newbie, so use the code above
at your risk

Please post your comments here, so that they are all in one place.