Title:  Writing A Lamson State Storage Backend

Earlier versions of Lamson assumed that you would use SQLAlchemy, and that you
wouldn't mind storing your state in the database using SQLAlchemy.  Well,
during the 0.9 redesign it became very easy to let you store the state however
and wherever you want.  In the new Lamson setup there is a bit more work to
create alternative storage, but Lamson comes with two default stores that you
can use to get started.

h2. The Default MemoryStore

When you get started with Lamson you'll definitely not want to go through the
trouble of setting up a custom store.  For your first days of development,
using the default
"MemoryStorage":http://lamsonproject.org/docs/api/lamson.routing.MemoryStorage-class.html
is the way to go.  After you get further in your development you want to switch
to the
"ShelveStorage":http://lamsonproject.org/docs/api/lamson.routing.ShelveStorage-class.html
to store the state in a simple Python
"shelve":http://docs.python.org/library/shelve.html store.

@MemoryStorage@ keeps the routing state in a simple dict in memory, and doesn't
provide any thread protection.  Its purpose is for developer testing and unit
test runs where keeping the state between disks is more of a pain than it is
worth.  Use the @MemoryStorage@ (which is the default) for your development
runs and for simple servers where the state does need to be maintained (very
rare). 

Here's the code to MemoryStore for you to just look at, it's already included in Lamson:

class MemoryStorage(StateStorage):
    """
    The default simplified storage for the Router to hold the states.  This
    should only be used in testing, as you'll lose all your contacts and their
    states if your server shutsdown.  It is also horribly NOT thread safe.
    """
    def __init__(self):
        self.states = {}

    def get(self, module, sender):
        key = self.key(module, sender)
        try:
            return self.states[key]
        except KeyError:
            self.set(module, sender, ROUTE_FIRST_STATE)
            return ROUTE_FIRST_STATE

    def set(self, module, sender, state):
        key = self.key(module, sender)
        self.states[key] = state

    def key(self, module, sender):
        return repr([module, sender])

    def clear(self):
        self.states.clear()


As you can see there isn't much to implement to make your own storage
for Lamson to use.

bq.  Keep in mind that this is just the storage *Lamson* needs to operate,
you probably don't want to be accessing this in your own application, 
and instead probably want to access the store you create yourself.

h2. The ShelveStorage

@ShelveStorage@ is used for your small deployments where you are mostly just
testing the deployment process or doing a small service.  It will store your
data between runs, and is probably fast enough for most sites, but you'll want
to ditch it if you ever:

# Run more than one process that needs the state information.
# Start to store everything in a database anyway.

The code to ShelveStorage (which is *already* part of Lamson) is more complex
since it must keep the threads happy, but you should read through it to get
an idea of how a more complex state store would work:

class ShelveStorage(MemoryStorage):
    """
    Uses Python's shelve to store the state of the Routers to disk rather than
    in memory like with MemoryStorage.  This will get you going on a small
    install if you need to persist your states (most likely), but if you 
    have a database, you'll need to write your own StateStorage that 
    uses your ORM or database to store.  Consider this an example.
    """
    def __init__(self, database_path):
        """Database path depends on the backing library use by Python's shelve."""
        self.database_path = database_path
        self.lock = threading.RLock()

    def get(self, module, sender):
        """
        This will lock the internal thread lock, and then retrieve from the
        shelf whatever module you request.  If the module is not found then it
        will set (atomically) to ROUTE_FIRST_STATE.
        """
        with self.lock:
            store = shelve.open(self.database_path)
            try:
                key = store[self.key(module, sender)]
            except KeyError:
                self.set(module, sender, ROUTE_FIRST_STATE)
                key = ROUTE_FIRST_STATE
            return key

    def set(self, module, sender, state):
        """
        Acquires the self.lock and then sets the requested state in the shelf.
        """
        with self.lock:
            store = shelve.open(self.database_path)
            store[self.key(module, sender)] = state
            store.close()

    def clear(self):
        """
        Primarily used in the debugging/unit testing process to make sure the
        states are clear.  In production this could be a bad thing.
        """
        with self.lock:
            store = shelve.open(self.database_path)
            store.clear()
            store.close()


h2. Using The ShelveStorage

You can use @ShelveStorage@ by simply adding this line to your @config/boot.py@ 
file just before you do anything else with the @Router@:

from lamson.routing import ShelveStorage
Router.STATE_STORE=ShelveStorage("run/states")

It actually doesn't matter currently when you do it, but it's good practice right now.

After you do that, restart lamson and it will start using the new store.
Notice that your *tests will not use this*.  It's not a good idea to have tests
use @ShelveStorage@, but if you want to turn it on for a run to see what
happens, then you can modify @config/testing.py@ the same way.  You could also
write a unit test that did this temporarily by putting that line in your test
case.

h2. What To Implement

If you want to implement your own then you just have to implement the methods
in
"StateStorage":http://lamsonproject.org/docs/api/lamson.routing.StateStorage-class.html
and make sure it behaves the same as MemoryStorage.  Look at the code to
ShelveStorage for a moment to see what you need:

class StateStorage(object):
    """
    The base storage class you need to implement for a custom storage
    system.
    """
    def get(self, module, sender):
        """
        You must implement this so that it returns a single string
        of either the state for this combination of arguments, OR
        the ROUTE_FIRST_STATE setting.
        """
        raise NotImplementedError("You have to implement a StateStorage.get.")

    def set(self, module, sender, state):
        """
        Set should take the given parameters and consistently set the state for 
        that combination such that when StateStorage.get is called it gives back
        the same setting.
        """
        raise NotImplementedError("You have to implement a StateStorage.set.")

    def clear(self):
        """
        This should clear ALL states, it is only used in unit testing, so you 
        can have it raise an exception if you want to make this safer.
        """
        raise NotImplementedError("You have to implement a StateStorage.clear for unit testing to work.")


There really isn't much to it, just methods to get and set based on the module
and sender's email address.  Also notice that you don't have to make it
readable in any complete sense, since Lamson doesn't do anything other than
get, set, and clear the state store (and it only clears on reloads and in
testing).

h2. Important Considerations

I am purposefully *not* telling you how to exactly implement it because I'm not
exactly sure what is needed as of the 0.9 release.  I use the the
ShelveStorage, but I'd like to hear what other people have written and then
start building infrastructure to make that easier.

There are some important things to consider when you implement your storage
though:

# Make sure that the calls to all methods are thread safe, and potentially process safe.
# If you do thread locking, use the *with* statement and an RLock.
# If your storage is potentially very slow, then consider a caching scheme inside, but *write that after making it work correctly.*
# Do *NOT* be tempted to store junk in this like it is a "session".  It should be lean and mean and only do state.
# Make sure you keep the key being used exactly as given.  You can seriously mess up Lamson's Router if you start getting fancy.

h2. Attaching It To The Router

Your storage backend will then be attached to the lamson.routing.Router in the
same way as what you did with ShelveStorage.  It really should be that simple
since the data stored in the state store is very minimal.

h2. Other Examples

If you want more examples then you can look at the @examples/librelist@ code to see
how "librelist.com":http://librelist.com/ uses "Django":http://www.djangoproject.com/
to store the state.
