If you’re not familiar with the concept of context-processors, I’ll also explain what those are and how they work, so don’t panic. Finally, i’ll also spend a minute explaining why I consider this to be best-practice.

What are template context processors?

Django’s context processors are a facility that allows you to provide data and callbacks to your templates.

You can do so in one of two ways:

1. On an individual request basis: by passing a custom Context value to your render_to_response() call
2. Globally: by creating a context processor method that accepts a HttpRequest object as input, and returns a payload or callback, then registering the context processor in your settings.py, then providing your render_to_response() call with the built-in RequestContext attribute instead of your own (you can always extend RequestContext to add more data on an individual request basis of course).

If that approach for passing data to templates sounded absurd and obfuscated to you, you’re not alone. The complexity involved in such a simple operation is unwarranted and counter-productive, but every system has its shortcomings.

The Issue

If you follow the Django documentation or The Django Book‘s approach to configuring your own custom context-processors, you’ll notice that you are encouraged to add to your settings.py a hardcoded list of built-in context processors. If you follow that approach, your context-processors declaration will look like this:

TEMPLATE_CONTEXT_PROCESSORS = ("django.contrib.auth.context_processors.auth",
"django.core.context_processors.debug",
"django.core.context_processors.i18n",
"django.core.context_processors.media",
"django.contrib.messages.context_processors.messages")

But if you pay close attention to the development version’s documentation, you’ll notice a couple of interesting notes:

Changed in Django Development version: “django.contrib.messages.context_processors.messages” was added to the default. For more information, see the messages documentation.
Changed in Django Development version: The auth context processor was moved in this release from its old location django.core.context_processors.auth to django.contrib.auth.context_processors.auth.

This is a red flag! The Django dev team would like you to use hardcoded values that reference to classes that will no longer exist once the next version is released, and that omit new processors that you are likely to require.

The Solution

Obviously Django has access to its own default settings, so there must be a way to simply extend the defaults instead of overriding them with hardcoded values. You just need to dig around a bit in the Django source code to find exactly how. I’ll save you some digging:

Add this at the top of your settings.py file:

import django.conf.global_settings as DEFAULT_SETTINGS

Then extend the default context processors:

TEMPLATE_CONTEXT_PROCESSORS = DEFAULT_SETTINGS.TEMPLATE_CONTEXT_PROCESSORS + (
"myapp.context_processors.example",
"myapp.context_processors.other_example",
# etc...
)

Why is this a best-practice approach?

In software-engineering, you want to ensure maximum (reasonable) future interoperability of all your components, and there is really no component as important as your actual development framework, Django or otherwise.

Using hardcoded values that are already defined somewhere not only breaks DRY, but it also introduces possible breakage on framework upgrades. I know you are diligent and always read the release notes before upgrading critical components, and I know you use a staging environment to test those changes, but by actively looking out for this kind of traps, you have just saved yourself some debugging time and some head-scratching.

Happy coding!

Share

Related posts:

1. Extending PostgreSQL with Python

CREATE PROCEDURAL LANGUAGE plpythonu;

Once the language bindings have been activated, you can start writing your stored procedures in python, however you should really read up on the following subjects first:

• CREATE FUNCTION statement syntax
• PL/Python Implementation Documentation

As an example, I’ve written a little SP in PL/Python to provide support for PCRE since the stock distribution of PostgreSQL only supports LIKE/SIMILAR, and POSIX Style Regular Expressions.

Let’s create our Python language binding, and create a standard text storage table:

-- Activate PL/Python
CREATE PROCEDURAL LANGUAGE plpythonu;
 
-- Create a plain text-storage table
CREATE TABLE text_storage
(
  id serial NOT NULL,
  payload character varying(128),
  CONSTRAINT text_storage_pkey PRIMARY KEY (id)
)
WITH (OIDS=FALSE);
ALTER TABLE text_storage OWNER TO xavier;
 
-- Let's throw in an index on the payload field for good measure
CREATE INDEX txt_payload_idx
  ON text_storage
  USING btree
  (payload);

Let’s now populate our table with some junk data:

INSERT INTO text_storage (payload) VALUES ('hello, world');
INSERT INTO text_storage (payload) VALUES ('the quick brown fox, blah blah blah');
INSERT INTO text_storage (payload) VALUES ('PCREs in Postgres');
INSERT INTO text_storage (payload) VALUES ('All hail Python!');
INSERT INTO text_storage (payload) VALUES ('Hello, test data!');
INSERT INTO text_storage (payload) VALUES ('Python would like to say Hello!');

And now we can go ahead and create our Python SP itself:

CREATE OR REPLACE FUNCTION pcre(text, text)
  RETURNS INTEGER AS
$BODY$import re
 
regex  = args[0]
in_str = args[1]
 
compiled = re.compile(regex)
 
IF compiled.search(in_str):
	RETURN 1
ELSE:
	RETURN 0$BODY$
  LANGUAGE 'plpythonu' VOLATILE
  COST 100;

As you can see, our PCRE matching system is extremely simple, yet pretty powerful. We import Python’s built-in re module, compile the specified regex argument, then attempt to match it against the other argument. Here’s a usage example on our test table:

SELECT id, payload FROM text_storage WHERE pcre('[H|h]ello', payload) = 1;
 id |             payload
----+---------------------------------
  1 | hello, world
  5 | Hello, test DATA!
  6 | Python would LIKE TO say Hello!
(3 rows)

As always, feel free to suggest any improvements.

Share

Related posts:

1. Fixing custom sequences in PostgreSQL
2. Ahh those cool little CLI tools…
3. Apache2 shortcuts

PostgreSQL will automatically create a sequence when you use the “SERIAL” datatype for your field, and will take care of assigning the default value of your field as the result of the nextval() call on the sequence, so most of the time you don’t need to interact directly with sequences.

Where things can get hairy however, is when you back up your data using pg_dump or any other mechanism, and restore that data in a table that is already populated. A common scenario for example, is populating a table that already has some recent data, with some older data you’ve been storing in archival. The opposite is true if you are trying to archive data from a table to a backup database for example.

Whenever you manually have to provide a value for the field assigned to a sequence, you are pretty much guaranteed to break the sequence unless you take the time to nextval() your sequence until it is in sync. This is a real problem, as pg_dump does not include sequence synchronization in its output.

The quickest way to synchronize a sequence, based on my observations, is to run the following query, taking care to replace the name of the sequence [SEQ] and the name of the associated table [TABLE] and field [FIELD]:

SELECT SETVAL([SEQ], COALESCE((SELECT [FIELD] FROM [TABLE] ORDER BY [FIELD] DESC LIMIT 1), 0)+1)

This will fetch the highest value of [FIELD] in [TABLE], increment it by 1, and synchronize the sequence [SEQ] to the new value.

I’ve also written the following little Python script that will look for any non-system sequence in the specified database, and use this method to repair it. Let me know if it works out for you, or if you’d like to suggest some improvements.

Requirements: Python 2.5+, PsycoPG2 Python module (python-psycopg2)

#!/usr/bin/env python
# -*- coding: utf-8 -*-
 
 
# Standard imports
import sys
import os
import time
from optparse import OptionParser
 
# psycopg2 import
try:
    import psycopg2
except ImportError, e:
    print 'You must install the python module named "psycopg2" in order to use this module.'
    sys.exit(os.EX_SOFTWARE)
 
 
 
class PgRepairman:
    def __init__(self, options, parser):
        self.options = options
        try:
            dsn = "dbname=%s host=%s user=%s" % (options.db, options.host, options.user)
            dsn += ("" != options.passwd) and ("password=%s" % options.password) or ""
            self.conn = psycopg2.connect(dsn)
            self.curs = self.conn.cursor()
        except Exception, e:
            print "ERROR - %s" % e
            sys.exit(1)
 
 
    # Returns a dict for a psycopg2 row object
    def _to_dict(self, desc, res):
        return dict(zip([x[0] for x in desc], res))
 
 
    # Attempt to locate all custom sequences
    def findSequences(self):
        seq_query = """
        SELECT pc1.relname AS seq, pc2.relname AS table, c.attname AS field
        FROM pg_depend, pg_class pc1, pg_class pc2, pg_attribute c
        WHERE pc1.oid = pg_depend.objid
            AND pc2.oid = pg_depend.refobjid
            AND c.attnum = pg_depend.refobjsubid
            AND c.attrelid = pc2.oid
            AND pc1.relkind = 'S'
            AND pc1.relname NOT LIKE 'pg_toast%%'
        """
 
        try:
            self.print_verbose(seq_query)
            self.curs.execute(seq_query)
        except Exception, e:
            print "[ERROR] - %s" % e
            sys.exit(1)
 
        desc = self.curs.description
        for row in self.curs.fetchall():
            yield self._to_dict(desc, row)
 
 
    # Increment the key value to the value of the sequence + 1
    def fixSequences(self):
        for seq in self.findSequences():
            print "Fixing sequence %s in table %s" % (seq['seq'], seq['table'])
            fix_query = "SELECT setval('%s', COALESCE((SELECT %s FROM %s ORDER BY %s DESC LIMIT 1), 0)+1)" % (seq['seq'], seq['field'], seq['table'], seq['field'])
            try:
                self.print_verbose(fix_query)
                self.curs.execute(fix_query)
            except Exception, e:
                print "[WARNING] - %s" % e
                pass
 
    def print_verbose(self, msg):
        if (True == self.options.verbose):
            print "[DEBUG] - %s" % msg
 
if __name__=='__main__':
    usage       = "Usage: %prog <options> [-v --verbose] [-u --username | -p --password \ -o --host | -d --database]"
    version     = "%prog v1.0\nDistributed under the LGPL2 License"
    description = "Increments all sequences in a PostgreSQL database"
    parser = OptionParser(usage=usage, version=version, description=description)
    parser.add_option("-v", "--verbose", action="store_true", dest="verbose", default=False, help="Enable extra output")
    parser.add_option("-o", "--host", action="store", dest="host", default="127.0.0.1", help="Database hostname/IP")
    parser.add_option("-u", "--username", action="store", dest="user", default="postgres", help="Database Username")
    parser.add_option("-p", "--password", action="store", dest="passwd", default="", help="Database Password")
    parser.add_option("-d", "--database", action="store", dest="db", default="template1", help="Database Name")
 
 
    try:
        (options, args) = parser.parse_args()
        obj = PgRepairman(options, parser)
        obj.fixSequences()
    except KeyboardInterrupt, e:
        sys.exit(1)

You can also download the script here.

Share

Related posts:

1. Extending PostgreSQL with Python
2. Apache2 shortcuts

To enable a module in your apache2 config, instead of doing the old

ln -sf /etc/apache2/mods_available/mod_rewrite.conf /etc/apache2/mods_enabled/mod_rewrite.conf
ln -sf /etc/apache2/mods_available/mod_rewrite.load /etc/apache2/mods_enabled/mod_rewrite.load

Next time, just do:

a2enmod rewrite && /etc/init.d/apache2 restart

To disable this module, try

a2dismod rewrite && /etc/init.d/apache2 restart

Similarly, to enable a site config, try

a2ensite myVhost.com && /etc/init.d/apache2 restart

and to disable it, obviously, try

a2dissite myVhost.com && /etc/init.d/apache2 restart

Finally, if you’d like to lint through your Apache2 config files before issuing a restart which might be responsible for some downtime if it doesn’t work, try

apache2ctl configtest

Share

Related posts:

1. Extending PostgreSQL with Python
2. Fixing custom sequences in PostgreSQL
3. Ahh those cool little CLI tools…