a suggestion for efficient and scalable counters in Datastore
As I’ve mentioned before, I’m trying to migrate urlBorg to Google AppEngine. urlBorg needs to count many things, like clicks on a short URL, etc, so I really need a scalable and efficient way to implement counters. This is not as trivial as it sounds in the Google AppEngine environment.
This post is actually the result of a good discussion done here
Here is the code I’ve come up with.
An example usage would be as simple as adding a line like (where page_id is a unique string identifying each page)
Acc(page_id).acc()
in each one of your pages. Getting the total coun is as simple as
Acc(page_id).val()
(Due to the way the total count is calculated, this may not give accurate results if you are in the middle of a traffic spike, but it’s good enough for web analytics usage)
class AccVals(db.Model):
cluster = db.StringProperty(required=True)
count = db.IntegerProperty(required=True)
updated = db.DateTimeProperty(auto_now=True)
rand = db.FloatProperty()
class Acc():
def __init__(self, name,init=0):
self.__sec = 0.1
self.__name = name
self.__init = init
def inc(self):
def trans(key):
obj = AccVals.get(key)
obj.count += 1
obj.put()
self.__val = obj.count
q = db.Query(AccVals).filter('cluster =',self.__name).filter('rand >', random.random()).get()
if (q):
if (datetime.datetime.now() - q.updated < datetime .timedelta(0,self.__sec)):
obj = AccVals(cluster=self.__name,count=self.__init, rand=random.random() )
key = obj.put()
else:
key = q.key()
else:
obj = AccVals(cluster=self.__name,count=self.__init, rand=1.0 )
key = obj.put()
db.run_in_transaction(trans,key)
return self.__val
def val(self):
total = 0
q = AccVals.all()
q.filter('cluster =',self.__name)
for r in q:
total += r.count
return total
It behaves relatively good and looks like it can scale no matter how
much traffic or traffic spikes you have.
If you look into it, you will see that a “counter instance” is chosen
in random. You may be tempted to use the “instance” that was updated
longer in the past ( order(’-updated’).get() ), but it turns out that
when you have a traffic spike (or whatever it is your counters count)
the indexes are not updated soon enough and this will return the last
records that were updated 
It looks like selecting a random
instance is no big deal in low traffic and works much better in high
traffic. I’ve also seen that after a while, you end up with the number
of counter instances that are required to handle the traffic of the
specific counter with few transaction collisions.
There is one interesting point: the value of self.__sec. I set it to
0.1 seconds, but this is just a value that looked good after some
tests. I have the impression that this value is *related* to some kind
of “global AppEngine constant”, measuring the time it takes for a
transaction to complete and safely propagate to the rest of the
infrastructure. I guess this varies, depending on the resource
allocation done for a specific app. Could someone from the AppEngine
development team give us some insight on this?
As I’ve mentioned before, I’m a Python newbie, so use the code above
at your risk
Please post your comments here, so that they are all in one place.
