googletorp.com

I've had a problem for some time with running tests on one of my older Django projects. In this project I had some models that wasn't being managed by Django. This mean that it was me who created the tables for the models instead of Django. This can be done pretty easily with the Meta class like this:

from django.db import models
class SomeModel(models.Model):
    ....

    class Meta:
        managed = False
        db_table = 'table_name'

“Now why would you do such a thing?” you might ask. I needed to create the tsvector data type - a postgres data type that is designed to support full text search. Something that quickly would be a pain with the Django ORM, that I enjoy to use. But in this case I was prepared to sacrifice the ORM to get full control over the SQL queries when I needed, to take full advantage of my postgres database.

Anyways everything worked out as planed and I didn't have any problems with my Django model etc. Then came the time to write some unittests and this is where I ran into some serious problems. When you use the Django testing framework, it will create a test database for you, to run you tests on. I discovered that unmanaged models wont get their tables created along with the others. This wasn’t a big surprise and would not be an issue, most of the time. In my case I needed those tables to be created before some of my apps got their tables created, since I had foreign keys to my unmanaged models. I thought such a task would be pretty simple, since this was an option that Django gives so easily and freely. I soon found out that is was not going to quite that easy to get my tables created. I was pretty fortunate that the unmanaged tables didn't have FK relations to some of the managed ones which would have made this much more complex.

So the task at hand was pretty simple. How to run some custom sql before the syncdb command is executed in test database creation.

I have listed here the steps I did to achive this, using django 1.1. In django 1.2 the test runner is a class instead of a function, but I bet the same procedure would work.

1 - Defining your own test runner

In the settings.py file, you can write a setting, if you want to overwrite the default test runner: django.test.simple.run_tests all you need to do is enter this in your settings.py:

TEST_RUNNER = 'path.to.run_tests'

What I did was to copy the default test runner and work with that.

2 - Digging into the code

In the default test runner the interesting thing for us happens at line 190 in simple.py:

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

create_test_db is a method on the BaseDatabaseCreation object defined in django.db.backends.creation. The code for the method looks like this:

def create_test_db(self, verbosity=1, autoclobber=False): """ Creates a test database, prompting the user for confirmation if the database already exists. Returns the name of the test database created. """ if verbosity >= 1: print "Creating test database..."

    test_database_name = self._create_test_db(verbosity, autoclobber)

    self.connection.close()
    settings.DATABASE_NAME = test_database_name
    self.connection.settings_dict["DATABASE_NAME"] = test_database_name
    can_rollback = self._rollback_works()
    settings.DATABASE_SUPPORTS_TRANSACTIONS = can_rollback
    self.connection.settings_dict["DATABASE_SUPPORTS_TRANSACTIONS"] = can_rollback

    call_command('syncdb', verbosity=verbosity, interactive=False)

    if settings.CACHE_BACKEND.startswith('db://'):
        from django.core.cache import parse_backend_uri
        _, cache_name, _ = parse_backend_uri(settings.CACHE_BACKEND)
        call_command('createcachetable', cache_name)

    # Get a cursor (even though we don't need one yet). This has
    # the side effect of initializing the test database.
    cursor = self.connection.cursor()

    return test_database_name

3 - Subclassing the DatabaseWrapper

I wanted add my own code before call_command('syncdb', verbosity=verbosity, interactive=False) was executed, as this is here Django creates the tables by calling syncdb on the freshly created test database. Since I knew what database type I have, I didn't need some logic to fetch the correct one as Django does, so I went right ahead and made a quick subclass of the psycopg2 DatabaseWrapper:

from django.db.backends.postgresql_psycopg2.base import DatabaseWrapper
class MyDatabaseWrapper(DatabaseWrapper):
def __init__(self, *args, **kwargs):
    super(FlexDatabaseWrapper, self).__init__(*args, **kwargs)
    self.creation = MyDatabaseCreation(self)

All I do is to change the DatabaseCreation object that is used, with my own, which I again needed to subclass:

class FlexDatabaseCreation(DatabaseCreation):
    def create_test_db(self, verbosity=1, autoclobber=False):
        ...

Then in my run_test I could now replace:

from django.db import connection
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

with

connection = MyDatabaseWrapper({
    'DATABASE_HOST': settings.DATABASE_HOST,
    'DATABASE_NAME': settings.DATABASE_NAME,
    'DATABASE_OPTIONS': settings.DATABASE_OPTIONS,
    'DATABASE_PASSWORD': settings.DATABASE_PASSWORD,
    'DATABASE_PORT': settings.DATABASE_PORT,
    'DATABASE_USER': settings.DATABASE_USER,
    'TIME_ZONE': settings.TIME_ZONE,
})
connection.creation.create_test_db(verbosity, autoclobber=not interactive)

4 - Making it all work

We now have control over the function that creates the test database with tables. But this also gave a little side problem. Because we don't use Django's connection, but our own, Django will keep on using it's regular db and not the test db. This is quite unfortunate at the result is running all the tests on a live / staging database. This is fixed with a few lines of code.

from django.db import connection
class FlexDatabaseCreation(DatabaseCreation):
    def create_test_db(self, verbosity=1, autoclobber=False):
        ...
        self.connection.settings_dict["DATABASE_NAME"] = test_database_name
        connection.settings_dict["DATABASE_NAME"] = test_database_name # New code.
        ...
        self.connection.settings_dict["DATABASE_SUPPORTS_TRANSACTIONS"] = can_rollback
        connection.settings_dict["DATABASE_SUPPORTS_TRANSACTIONS"] = can_rollback # New code.
        ...

With this little addition we now have created a test database and make Django use it, we are really where we started, only now Django is using our code to do it instead. The last thing that is needed is to add the custom sql. I did it be reading a file where I have the SQL for the tables stored:

from django.db import connection
class FlexDatabaseCreation(DatabaseCreation):
    def create_test_db(self, verbosity=1, autoclobber=False):
        ...
        f = open('path/to/schema.sql')
        sql = f.read()
        f.close()
        cursor.execute(sql)
        cursor.connection.commit()

        call_command('syncdb', verbosity=verbosity, interactive=False)

5 - Success

That's all there is too it. I would have loved if there was a signal pre syncdb, so you simply could setup the SQL to be run in a signal instead of having to grind through all this code. But in the end, it was a fun challenge to overcome and not a type of thing I’ve done much when it comes to Django.

A field formatter is what CCK uses to format it's fields or in Drupal terms: theme. All that a formatter really is, is a theme function that's designed for CCK. Where ever a field is displayed it can have several options as to how it should be displayed. One example could be link fields which by default can be displayed in different ways, title as link, URL as link etc. This can be selected as standard for the node full and teaser views, but can also be used in different views where you thus can display the same field in different ways.

The hardest thing about making a formatter for fields, is finding out how you make something that CCK can understand and will know about, the rest is pretty much like any other theme function. The tricky part here is that CCK doesn't have that much documentation, which makes finding out how to do these things more difficult. The way I figured out how to do what I needed, was mostly looking at how others did it in various modules. So this will be a quick 3 step walkthrough of what I've find during my search and experimentation.

Step 1

What you need to do first, is to make an inplementation of hook_field_formatter_info. This is what tells CCK that your formatter exists. That way CCK will present it as a display option where needed. In code this looks like this.

function module_name_field_formatter_info() {
    $item = array();
    $item['formatter_name'] = array(
        'label' => t('The label which is what will be shown in the AI'),
        'field types' => array('link'), // an array of field types where for which the formatter can be used
        'multiple values' => CONTENT_HANDLE_CORE,
     );
     return $items;
}

Label and field types are pretty easy to understand. At the time of writing I can't remember how multiple values work exactly. It was something I didn't need to explore as my fields didn't have multiple values. But I believe that you can decide if cck or your something else should determine how multiple values should be handled. Maybe wanted to make 1 table with all the values instead of 1 table for each value or something similar.

In the example above I used the field type link, which is for link fields. This was what I needed to make a formatter for, since the site I was building was selling clicks. A lot of tracking was also involved so being able to control the output of the link was very important. All the links was external of cause, but to be able to do tracking and such we wanted to create an internal URL instead. We could then react on that, store useful information we needed and redirect to the actual url. This is just a simple use case, of how controlling the output can be very important and that the actual output you generate can be very different from what is "normal".

Step 2

Amyways once you have made your implementation of hook_field_formatter_info, the rest is pretty easy.

Implementation of hook_theme, is the next step, which is pretty much like usual, except you need to add formatter to the theme name so you get a theme name that looks like A_formatter_B, where A is the module name and B is the formatter name:

function module_name_theme() {
    $items =  array();
    $items['module_name_formatter_formatter_name'] = array(
        'arguments' => array('element' => NULL),
    );
    return $items;

Step 3

The only thing that's left if to create the actual theme function, which is just like you would expect. You just prepend _theme to the name.

function  theme_module_name_formatter_formatter_name($element) {
     // Do your stuff here.
}

One thing that you need to be aware of, is that $element will vary a bit in different cases. These are not big chances, but you should be aware that there can be changes, especially when you use the formatter in a normal node display vs using it for a field in views.

That's all there is to it, you can now create your own CCK field formatters.

Making a jQuery plugin is actually not so difficult that I once thought it was. One thing you need to learn when doing this is closure. If you plan on doing any real JavaScript development, it's time well spent to wrap your hear around this.

Closures

Closures is a technique that's very powerful. The thing about JavaScript, is that there is no name-spaces, all of the code and variables are run in the browser's document. This can give some nasty problems, because you when writing a plugin, you don't know the context where it'll be used. To make it of any use, you have to be certain that you don't overwrite global variables like the $. The way to overcome this is closure. You wrap a function inside parentheses making it a function and consequently invoke it. Doing that, you actually wrap your function inside what's called a lambda function. The tricky part is that you can invoke your lambda function with some params, and they will be used inside your anonymous function. Doing this, you are able to create a memory bubble that wont effect the global name-space. This sounds a bit more harry than it actual is so let's look at the code.

arg1 = 'test';
(function(arg1, arg2) {
    // arg1 will have the value of param 1
    //  arg 2 will be param2
})(param1, param2)
// arg1 is still 'test'

So using this syntax, we declare 2 functions, one inside the other and execute both. what's really handy is that arg1 will not conflict with the global name-space. Changes to arg1 inside the function will manifest as changes to param1. In jQuery plugins you usually want to do it like this:

(function($) {
    // ...
})(jQuery);

This will enable us to make code that uses the $ for jQuery, even if the $ is used for something else like prototype outside the scope of the function. The nice thing with this setup, is that we can now extend jQuery inside our function, and it will be manifested to the global jQuery variable. But because JavaScript has a functional scope, all the functions and variables we define inside the anonymous function will be available to the plugin we make, but hidden to the outside. This is one of the very fun things about JavaScript as a language.

After this long rant on closures, we're almost done. jQuery provides us with function we can use to expand the jQuery name-space, creating our own function that can be invoked with .functionName(). Adding this to our code block we get:

(function($) {
    $.fn.extend({
        plugin: function() {
        // ...
        }
    });
})(jQuery);

This will register the function name plugin that would be called doing something like this $("h2").plugin();

All that is left now, is to actually do the plugin. Another thing worth noting, is that inside the function, this will refer to result of the jQuery selection. This means that what we get already have jQuery attached to it. An example of a small plugin we could make ourselves, it the live binding of the click event. jQuery already has a short hand for the normal binding of a click event .bind("click" fn). But we could make our own for the live version, this is pretty easy and would look like this:

(function($) {
    $.fn.extend({
        lclick: function(fn) {
            // Make sure to only make the binding if the arg is a function.
            if (typeof fn == "function") {
                this.live("click", fn);
            }
            return this;
        }
    });
})(jQuery);

With the code above added, we are now able to do .click(fn) for the normal bind and .lclick(fn) for the live binding of the click event. If we wanted to we could also add the execution of the event when no params are presented. That could be done as an if statement when fn is undefined. Another thing to note, is that I am returning this in the lclick function. This is to allow us to chain the events, so we can do $(selector).lclick().show()... which is one of the very nice things about jQuery.

This "plugin" is not as powerful as many of the very nice jQuery plugins that have been made. But I hope I have illustrated that making your own plugin, isn't that difficult, and can actually be done quite easily. Sometimes you might want to convert some of your usual javascript function into plugin, if you tend to use that block of code a lot. It'll also make it easier to manage your code afterward.

Good luck with those plugins!

About a year ago I started learning python, after having stopped studying chemistry at the university of Copenhagen. A good friend of mine had started his own company, Reveal IT and I joined him as what you could call an apprentice. After getting the basics of python I started right away with Django. I can still remember the struggles I had in the beginning understanding stuff like HttpResponse, templates etc. Luckily he had a good learning project for me to work on, Djime, what was to be a Django based web app time tracker.

Much has happened the past year, I've not only learned Python and Django, but also HTML, CSS, JavaScript - jQuery, PHP, Drupal. I now spend most of my time working with Drupal, making websites for our clients. Something I never imagined myself doing, and having a lot of fun doing it. But I must admit, Django/Python is still very special to me being my first language and framework. Though I don't often have time to work on Djime anymore, I recently had some extra time which was spent on a re-factor of the Djime codebase. Under the codename Pine, we now have a working prototype of Djime integrated with Pinax. You can take a look at the code, at my djime github repo.

So doing all these fun things on the web, I decided it was about time for me to actually have a blog. I'll probably focus on writing about some of the things I do with the web, conquests, defeats and all that. Hopefully to help others, but if nothing more, then at least it can stand for me as a reminder.

But enough of this ranting, it's time for this, my first blog post ever to be published. On a side note, the site here will be evolving slowly, while I try to figure out some styles for this site.