One easy way to lower the cost of URL-shortening is to use our own domain names in place of tinyurl.xom, bit.ly, tr.im, et al. Any one of those services could take the lead here by allowing for that. Let me map my own domain onto theirs, easily back up all my data, and give me the ability to switch services when I want, or when I need to.

I would call it “data protability for URL shorteners”. It’s what I’ve been trying to push for more than a year by urlborg.xml (and was actually inspired by an older post of Dave Winer) .

From now on urlBorg will ask you to login if you want to create a short URL. The reason is that spammers took advantage of the URL shortening service to create short URLs they included in spam emails. As a result I got many complaints, I even have to pay $199 to GoDaddy (where the domain is registered) to

[...] cover the costs of responding to or “cleaning up” the outstanding spam complaints Go Daddy has received against your domain name so far. This fee is also used to cover the costs of processing future complaints, forwarding these complaints to you, and ensuring that the offending affiliates have been deleted. [...]

I still think GoDaddy is wrong to consider urlBorg part of the problem, but the truth is that even free on-line services should try to make the web a better and safer place. Authentication should make urlBorg spammer-unfriendly, or at least make it easier for me to fight them by disabling spam accounts and all their short URLs if necessary.

Requiring an account might be annoying to some, I understand. On the other hand, you can be sure all your short URLs are tracked and added to your profile (a feature that most urlBorg users like and depend on). Plus it will add some more “credibility” to any links starting with “ub0.cc”.

Obviously, users just clicking on short URLs are not affected, this is for people using urlBorg to create short URLs.

API usage is not affected, either.

This is one of the main design decisions I had to do when I started developnig urlBorg. And I’ve changed my mind a couple of times. But I think that creating a new short URL is the right way to go. Here is why.

The creation of a short URL is a “gesture of attention” (if you are not familiar with the term, read Geting a piece of the action: The attention economy). An if a URL shortener is worth something, this is because it is a good database of two kind of “gestures”:
- the one is when someone manifests his interest on a specific URL, by creating a short URL for it.
- the other is when someone clicks on the short URL.

urlBorg keeps track of both. The click stats are visible to anyone, the just have to add /i at the end of a urlBorg short URL, like this http://vrypan.net/log/?p=420 read more →]]> summize.com is a search engine for twitter -and a great one!

urlBorg now queries summize.com for each short URL and displays the results in the preview page. This way it is easier to guess or understand what the link is about, before going to the actual page.

urlBorg lets users see more detailed stats now.

There are many ways to access the click stats:

1. Click the “more info” link at the bottom of any “preview page”
2. Login to your account and click the short URL on any of the “created” or “clicked” links.
3. append /i at the end of a short url. ex ub0.cc/6/0h/i

The graphs are created using the Google Chart API and IMHO the look quite nice

One of the best url shortening tools – jim.hellas – May 24, 2008
UrlBorg has a very simple interface, provides personal account and helps you keep track of your urls. Simply the best

The best out there! – kostis_at_quatzacoalt.com – May 24, 2008
My choice for URL shortening! Useful features! I’m sure more are on the way!

Great stuff – stefanos.kofopoulos – May 24, 2008
…now running on scalable environment.

urlBorg rules! – niels.vaneck – May 24, 2008
Scalable url shortening in combination with user accounts makes for a very useful service.

Wow! Thank you!

1. It will keep a record of all the “short URLs” you’ve created, with stats (number of clicks). This is a nice way to keep track of the URLs you’ve created but also to see how many clicks each one of them had.

2. It will keep a record of all urlBorg short URLs you’ve clicked on. This is an optional feature, you have to enable it explicitly. I often find myself looking for “that link to a funny video someone posted on twitter yesterday”. Well if this someone used urlBorg to create short links, now I will find it in my history.

To use both of them, you have to login of course. But I took advantage of AppEngine’s Users API which means that if you already have a GMail account you login using it -no registration, etc. (I know I should have used OpenID, but for now it was so much simpler to use Google’s authentication -but OpenID is on the ToDo list).

Now, if someone could advice me on what is the right format to export the attention data gathered, so that users are able to reuse them and move them around, please let me know.

You will be redirected to your urlBorg account page, where you will se the option “enable previews”. Previews are simple pages that look like this (here, I take advantage of a hidden urlBorg feature: if you add /p at the end of a short URL you “force” the preview to show up). Their obvious value is to protect you from going to pages you wouldn’t like to visit, but their URL is hidden by urlBorg.

I think “previews” can be much more. Here are a couple of features I already implemented.

MP3 files
If a URL goes to an MP3 file (determined by the MIME type reported by the hosting server), the preview will also show you flash mp3 player that allow you to listen to it. Here is an example.

maps
For this one, I have to thank Raphael from U.[lik]. Raphael is has been using urlBorg to share links to Google Maps (those links can be long and ugly). Now, if you link to a page in Google Maps or Yahoo! Maps, the preview will also display a link to the other service, where you will se the same map.

Here is a Google Maps expample (again, note the /p at the end of the URL) and a Yahoo! Maps example. You don’t have to do something special. urlBorg will examine the structure of the link and display automagicaly the “extra info”.

So, what do you think?

But why build “yet another URL shortening service”, when it’s so easy to build one? Any web developer could build one in less than an hour, couldn’t they?

The truth is that building a URL shortening service is a trivial task. Building one that could scale is not. I designed urlBorg having in mind “will it work if it made it to TechCrunch or if CNN.com made extensive use of it?” Building such a service is not trivial, believe me. (And I won’t know if urlBorg will make it either, but I think it will.)

But scale wasn’t my only motivation. I believe there’s a lot of space to add value to such a simple service. A quick look at the API will reveal some of my ideas -urlBorg goes beyond returning a short URL.

More details to come soon

This post is actually the result of a good discussion done here

Here is the code I’ve come up with.
An example usage would be as simple as adding a line like (where page_id is a unique string identifying each page)

Acc(page_id).acc()

in each one of your pages. Getting the total coun is as simple as

Acc(page_id).val()

(Due to the way the total count is calculated, this may not give accurate results if you are in the middle of a traffic spike, but it’s good enough for web analytics usage)

class AccVals(db.Model):
       cluster = db.StringProperty(required=True)
       count = db.IntegerProperty(required=True)
       updated = db.DateTimeProperty(auto_now=True)
       rand = db.FloatProperty()

class Acc():
       def __init__(self, name,init=0):
               self.__sec = 0.1
               self.__name = name
               self.__init = init

       def inc(self):
               def trans(key):
                       obj = AccVals.get(key)
                       obj.count += 1
                       obj.put()
                       self.__val = obj.count

               q = db.Query(AccVals).filter('cluster =',self.__name).filter('rand >', random.random()).get()
               if (q):
                       if (datetime.datetime.now() - q.updated < datetime .timedelta(0,self.__sec)):
                               obj = AccVals(cluster=self.__name,count=self.__init, rand=random.random() )
                               key = obj.put()
                       else:
                               key = q.key()
               else:
                       obj = AccVals(cluster=self.__name,count=self.__init, rand=1.0 )
                       key = obj.put()

               db.run_in_transaction(trans,key)
               return self.__val

       def val(self):
               total = 0
               q = AccVals.all()
               q.filter('cluster =',self.__name)
               for r in q:
                       total += r.count
               return total

It behaves relatively good and looks like it can scale no matter how
much traffic or traffic spikes you have.

If you look into it, you will see that a “counter instance” is chosen
in random. You may be tempted to use the “instance” that was updated
longer in the past ( order(‘-updated’).get() ), but it turns out that
when you have a traffic spike (or whatever it is your counters count)
the indexes are not updated soon enough and this will return the last
records that were updated It looks like selecting a random
instance is no big deal in low traffic and works much better in high
traffic. I’ve also seen that after a while, you end up with the number
of counter instances that are required to handle the traffic of the
specific counter with few transaction collisions.

There is one interesting point: the value of self.__sec. I set it to
0.1 seconds, but this is just a value that looked good after some
tests. I have the impression that this value is *related* to some kind
of “global AppEngine constant”, measuring the time it takes for a
transaction to complete and safely propagate to the rest of the
infrastructure. I guess this varies, depending on the resource
allocation done for a specific app. Could someone from the AppEngine
development team give us some insight on this?

As I’ve mentioned before, I’m a Python newbie, so use the code above
at your risk

Please post your comments here, so that they are all in one place.

I mean, creating short URLs is a trivial thing. If you want to make a service that stands out, it has to be that it takes care of the little details in a much better way than the rest. And you have to be sure that your service will be able to scale.

So, moving urlBorg to Google App Engine should be a no brainer, right? Wrong.

My main issue is AppEngine Datastore.

The App Engine datastore is not a relational database. While the datastore interface has many of the same features of traditional databases, the datastore’s unique characteristics imply a different way of designing and managing data to take advantage of the ability to scale automatically.

So, forget about queries involving group functions like count(*), min(), max()…

I wish they had some good examples on how to use the AppEngine Datastore to do data mining. How should/would a “web analytics” application be implemented using AppEngine for example?

On the urlborg.xml front, a lot has been going on during the last days. A new interface, documentation, etc. Check out urlBorg developers blog for more.

I chose the Prologue wordpress theme that resembles twitter a lot in look and feel. I also installed Twitter Tools and set it up to update twitter.com/urlborg. I like the way those too integrate together.

Here are a couple of entries from my logs:

grep "urlBorg/1.0" < access_log | grep -v HEAD 

69.73.152.127 - - [14/Feb/2008:16:48:03 +0200] "GET /weblog/ HTTP/1.1" 200 26385 "-" "urlBorg/1.0 (+http://www.urlb.org/) [lgfy]"
69.73.152.127 - - [14/Feb/2008:16:48:20 +0200] "GET /weblog/2007/12/08/3214/ HTTP/1.1" 200 19345 "-" "urlBorg/1.0 (+http://www.urlb.org/) [lggw]"

Not only I am able to see when a shortcut was created to one of my pages, but I can also see the “URL key” (ex. “lgfy”) -using the API I could get extra info about the short URL, like stats, etc.

Download

First download urlborg.php (and, of course, make sure you have your API key).

Now, creating short URLs and getting info about them is easy. Here is an example:

require_once 'urlborg.php' ;
$ub = new urlBorg('use your API key here') ;
$info = $ub->get_info('lgfy') ;
if ($ub->get_error()) {
   echo 'ERROR! Code:' . $ub->get_error('n') .
   ' desc:' . $ub->get_error('s') ;
} else {
   print_r($info) ;
}
$short = $ub->create_new('http://vrypan.net/') ;
if ($ub->get_error()) {
    echo 'ERROR! Code:' . $ub->get_error('n') .
    ' desc:' . $ub->get_error('s') ;
} else {
    print_r($short) ;
}

(the example is included in the class headers. Don’t copy-paste it from this post since WordPress seems to change the quotes in ways that PHP doesn’t like

• API calls “create”, “info” will return target url mime type and size
• urlkeys are unique for each url (and API key). If you are using the web interface, this means that next time someone submits the same URL they will get the same short URL. If you are using the API, you will get a unique short URL for each pair of (url,apikey)
• the “send SMS” option has been removed since it was inactive anyway

More info on the API can be found here.

Now, you can subscribe to search results (RSS feed)!