Remember my last major blog engine upgrade? The one where I deleted my database and had to piece it together from the Google cache after that? And that's after spending a week rewriting my blog from the bottom up to be more modular and stable. Well, it turns out that as time passes, what I did back then wasn't enough.

I had a few problems. The first, the Durus object database I was using was fairly unreliable. From time to time, it would randomly crash and burn for no reason. I'd have to log in and restart both it and the WSGI server to get it running again. Plus, if any of my discerning web developer readers (cue cricket chirps) observed it, my server lagged 2-5 seconds on every request.

Well, guess what?

Also, chicken butt.

This means I got my plan to overhaul my blog's storage module to completion. The first (and most major change) was to switch from my Durus database to a ZODB (Zope Object DataBase) system. The decision to do this was made long ago, and the main reasons were as follows:

• Durus is a "light" rewrite of ZODB, lacking some of its features, but being "good enough" for most uses
• ZODB has rollback (undo) support
• ZODB has a distributable structure, so in high load times (if I get Slashdotted again) I can distribute database load across multiple computers.
• ZODB won't crash randomly on me.
• ZODB is actively developed by the Zope community, who I have more experience with and trust in more than the Durus folks.

Why I did not switch (back) to a relational database like MySQL, I will discuss later.

Along with my switch to ZODB, I made my data system much more transparent. Previously, a piece of routine database work looked like this:

from storage import newDurusConnection
conn, root = newDurusConnection()
del root['entries']['some-entry'].comments[3]
conn.commit()

True fact: it gives me some tremendous personal satisfaction to do that. However, the duplicate objects I need to take care of is sort of inconvenient. Plus, it means that when I do work outside of my storage.py file, it means that the rest of my blog needs to be aware of what type of database I'm using, in order to know how to commit it... which betrays my "modularity" for my blog! A much better solution is:

from storage import StorageFactory
store = StorageFactory()
del store['entries']['some-entry'].comments[3]

One line shorter, and looks more straightforward. I don't even have to concern myself with the connection! This works because of a little wrapper class I made:

class BlogStorage:
    # ...
    def __init__(self, zaddr = ZEO_ADDR, zport = ZEO_PORT, conn = None):
        self.conn = conn
        if not conn:
            self.storage = ClientStorage((zaddr, zport))
            self.db = DB(self.storage)
            self.conn = self.db.open()
        self.root = self.conn.root()

    def __del__(self):
        transaction.commit()
        persistconn[persistconn.index('taken')] = self.conn

    def __getitem__(self, key):
        return self.root[key]

    def __setitem__(self, key, val):
        self.root[key] = val

    def keys(self): return self.root.keys()

    def items(self): return self.root.items()

    def values(self): return self.root.values()

    def has_key(self, key): return self.root.has_key(key)

What this does is, it either establishes a new connection, or works with a given one, and acts as a transparent dictionary-like object on top of the "root" object at the base of my storage. You can see that in the __del__ method, the transaction (and all changes) are committed. This method is the destructor of the class, which is called whenever all references to an instance drop. It also places the connection object back on the "persistconn" list of available connections.

In this way, I accomplish two things: transparency of committing objects, and database connection recycling. The latter solves the problem of the lag I mentioned earlier. Pages were slow in loading because a connection to a Durus or ZODB database takes a few seconds to set up, and since each HTTP request (of which there can be many) needs at least one, this caused significant slowdowns.

If you're curious about what else I changed, feel free to take a look at the source code.

In another train of thought, I am considering not giving my entire blog entries to feed readers anymore.

Aw hell naw!

Rather, I would give the first paragraph (or so), along with a "Read more..." link. Not only is some content best viewed on my site (such as images with transparency), but it would also promote commenting, get more of your comments read, and last but not least, raise my reader count!

So, look forward to that!