As usual, I’ve also provided a unit-test suite for each algorithm.

Please note: This is only intended to demonstrate the algorithms. If you are actually trying to do some real work with binary numbers in Python, note that the language itself provides a highly optimized implementation of binary numbers and related operations. Here are a few examples:

# Decimal to binary conversion:
>>> bin(15)
'0b1111'
 
# Binary to decimal conversion:
>>> str(0b1111)
'15'
 
# Binary addition:
>>> bin(0b1111 + 0b10011)
'0b100010'
 
# Binary multiplication:
>>> bin(0b1111 * 0b10011)
'0b100011101'

Binary Addition

We will use strings in this example to represent sequences of bits. The algorithm operates right-to-left, maintaining a carry each time it adds two 1-valued bits.

Code:

"""Simple binary operation algorithm."""
 
import math
 
# a and b should be string representation of the binary components
# You could easily extend this to overload the + operator for binaries,
# but this is already built into python.
def binary_addition(bin_a, bin_b):
    """Performs a binary addition against two provided binary numbers"""
 
    # Pad the components so they are of equal length
    max_length = max(len(bin_a), len(bin_b))
    bin_a, bin_b = bin_a.zfill(max_length), bin_b.zfill(max_length)
 
 
    carry = 0
    result = ''
 
    # Note that we work from right to left
    for i in range(0, max_length)[::-1]:
        tmp = int(math.floor((int(bin_a[i]) + int(bin_b[i]) + carry)/2))
        res = str(int(bin_a[i]) + int(bin_b[i]) + carry - 2*tmp)
        result += res
        carry = tmp
 
    result = (result + str(carry))[::-1]
    try:
        return result[result.index('1'):]
    except ValueError, ex:
        return '0'

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_addition import binary_addition
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary addition against our own"""
 
    def test_add_same_length(self):
 
        """Adding 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """ B has more digits than A """
 
        bin_a = '01000101'
        bin_b = '1110010110101011101101011101000101'
 
        self._compare_additions(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """B has less digits than A"""
 
 
        bin_a = '1110001001001001001101101011000101'
        bin_b = '0110010'
 
        self._compare_additions(bin_a, bin_b)
 
    def _compare_additions(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_add = binary_addition(bin_a, bin_b)
        py_add = bin(eval("0b%s" % bin_a) + eval("0b%s" % bin_b))
 
        algo_res = bin_add[bin_add.index('1'):]
        py_res = py_add[py_add.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Binary Multiplication

We are still using strings to represent bit sequences. We are still working from right to left, shifting to the left every number from the first number that needs to be multiplied by 1 in the second number. We then append the result to a list that we sum at the end, using the previous algorithm.

Code:

"""Simple Binary multiplication algorithm"""
 
 
import sys
import os.path
 
from binary_addition import binary_addition
 
 
def binary_multiplication(bin_a, bin_b):
    """Multiplies two binary numbers by using python lists to
    easily shift digits around"""
 
    temp_result = []
    result = "0"
 
    # Remove any unnecessary padding
    bin_a = bin_a[bin_a.index('1'):]
    bin_b = bin_b[bin_b.index('1'):]
 
    for i in range(0, len(bin_b))[::-1]:
        if bin_b[i] == '1':
            temp_result.append("%s%s" % (bin_a, "0" * (len(bin_b)-i-1)))
        else:
            temp_result.append("0")
 
 
    for val in temp_result:
        result = binary_addition(str(result), str(val))
 
    return result

Unit Test:

"""Unit test for binary_addition.py"""
 
import unittest
from binary_multiplication import binary_multiplication
 
class TestBinaryAddition(unittest.TestCase):
 
    """Comparing python binary multiplication against our own"""
 
    def test_add_same_length(self):
 
        """Multiplying 2 binary numbers of the same length"""
 
        bin_a = '001001101'
        bin_b = '011011010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_simple(self):
 
        """Multiplying 2 simple binary numbers"""
        bin_a = '110'
        bin_b = '101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_larger(self):
 
        """B has more digits than A"""
 
        bin_a = '010'
        bin_b = '111001011000101'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def test_add_b_smaller(self):
 
        """ B has less digits than A """
 
        bin_a = '111001011000101'
        bin_b = '010'
 
        self._compare_multiplications(bin_a, bin_b)
 
    def _compare_multiplications(self, bin_a, bin_b):
 
        """Compares the python implementation against ours"""
 
        bin_mult = binary_multiplication(bin_a, bin_b)
        py_mult = bin(eval("0b%s" % bin_a) * eval("0b%s" % bin_b))
 
        algo_res = bin_mult[bin_mult.index('1'):]
        py_res = py_mult[py_mult.index('1'):]
 
        self.assertEqual(py_res, algo_res)
 
 
 
if '__main__' == __name__:
    unittest.main()

Share

Related posts:

1. Algorithms in Python: Base Expansion
2. Extending PostgreSQL with Python

This week, we’ll cover an introductory algorithm for converting from one numeral base to another. Here is the python code:

"""Simple implementation of a base expansion algorithm"""
 
import math
 
def base_expand(base, val):
    """This simple function performs a base-expansion from decimal
    using moduli and a translation table. The translation table is
    a clear limitation here, in that it implies the maximum base
    is 36."""
 
    if (base < 2) or (base > 36):
        raise BaseOutOfBoundsError(base)
 
    trans_table = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'
    res = ''
 
    while val != 0:
        res += trans_table[(int(val % base))]
        val = math.floor(val / base)
    return res[::-1]
 
class BaseOutOfBoundsError(Exception):
    """Base must be between 2 and 36"""
    def __init__(self, val):
        self.val = val
    def __str__(self):
        print "\nInvalid base: %s. Base must be (x | x > 1; x < 37)" % \
            self.val

Here is a unit-test for base_expand.py:

"""Unit tests for base_expansion.py"""
 
import unittest
from base_expansion import base_expand, BaseOutOfBoundsError
 
# Unit tests for base_expansion.py
class TestVals(unittest.TestCase):
    """Test suite"""
 
    def test_known_values(self):
        """Testing against known values"""
 
        vals = [{'val': 8, 'base': 2, 'expect': '1000'},
            {'val': 915652, 'base': 16, 'expect': 'DF8C4'},
            {'val': 256, 'base': 10, 'expect': '256'},
            {'val': 88189, 'base': 8, 'expect': '254175'}]
 
        for val in vals:
            self.assertEqual(val['expect'], \
                base_expand(val['base'], val['val']))
 
    def test_invalid_base(self):
        """ Testing invalid bases """
 
        bases = [1, 37, 50, 100]
        for base in bases:
            self.assertRaises(BaseOutOfBoundsError, base_expand, \
                base, 1)
 
 
 
if '__main__' == __name__:
    unittest.main()

Share

Related posts:

1. Algorithms in Python: Binary Operations

If you’re not familiar with the concept of context-processors, I’ll also explain what those are and how they work, so don’t panic. Finally, i’ll also spend a minute explaining why I consider this to be best-practice.

What are template context processors?

Django’s context processors are a facility that allows you to provide data and callbacks to your templates.

You can do so in one of two ways:

1. On an individual request basis: by passing a custom Context value to your render_to_response() call
2. Globally: by creating a context processor method that accepts a HttpRequest object as input, and returns a payload or callback, then registering the context processor in your settings.py, then providing your render_to_response() call with the built-in RequestContext attribute instead of your own (you can always extend RequestContext to add more data on an individual request basis of course).

If that approach for passing data to templates sounded absurd and obfuscated to you, you’re not alone. The complexity involved in such a simple operation is unwarranted and counter-productive, but every system has its shortcomings.

The Issue

If you follow the Django documentation or The Django Book‘s approach to configuring your own custom context-processors, you’ll notice that you are encouraged to add to your settings.py a hardcoded list of built-in context processors. If you follow that approach, your context-processors declaration will look like this:

TEMPLATE_CONTEXT_PROCESSORS = ("django.contrib.auth.context_processors.auth",
"django.core.context_processors.debug",
"django.core.context_processors.i18n",
"django.core.context_processors.media",
"django.contrib.messages.context_processors.messages")

But if you pay close attention to the development version’s documentation, you’ll notice a couple of interesting notes:

Changed in Django Development version: “django.contrib.messages.context_processors.messages” was added to the default. For more information, see the messages documentation.
Changed in Django Development version: The auth context processor was moved in this release from its old location django.core.context_processors.auth to django.contrib.auth.context_processors.auth.

This is a red flag! The Django dev team would like you to use hardcoded values that reference to classes that will no longer exist once the next version is released, and that omit new processors that you are likely to require.

The Solution

Obviously Django has access to its own default settings, so there must be a way to simply extend the defaults instead of overriding them with hardcoded values. You just need to dig around a bit in the Django source code to find exactly how. I’ll save you some digging:

Add this at the top of your settings.py file:

import django.conf.global_settings as DEFAULT_SETTINGS

Then extend the default context processors:

TEMPLATE_CONTEXT_PROCESSORS = DEFAULT_SETTINGS.TEMPLATE_CONTEXT_PROCESSORS + (
"myapp.context_processors.example",
"myapp.context_processors.other_example",
# etc...
)

Why is this a best-practice approach?

In software-engineering, you want to ensure maximum (reasonable) future interoperability of all your components, and there is really no component as important as your actual development framework, Django or otherwise.

Using hardcoded values that are already defined somewhere not only breaks DRY, but it also introduces possible breakage on framework upgrades. I know you are diligent and always read the release notes before upgrading critical components, and I know you use a staging environment to test those changes, but by actively looking out for this kind of traps, you have just saved yourself some debugging time and some head-scratching.

Happy coding!

Share

Related posts:

1. Extending PostgreSQL with Python

I didn’t get around to sit down and come up with a write-up about the experience as i’ve been working on other projects every evening this week, but I finally have a minute to do so now.

As far as the convention event itself went, there is definitely a lot of good and a few notable logistical oversights that are worth mentioning, but i’m not really interested in addressing either on here, as I’m really more interested in sharing my impression of the technical panels I’ve attended.

I have to say, I’m very excited about every single one of the panels I’ve been able to attend this year. They were all highly informative, and the speakers very motivated and passionate about their material. Here’s a short summary of the events I was especially excited about:

• Neural Networks
  This panel was held by Dr. Stanley C. Mortel. It explained the basic concepts behind the idea of building and training neural computation networks. It was a very abstract fly-by course, which I feel is a very appropriate way to introduce this type of material. There was a second part of this panel available the next day but we weren’t fortunate enough to attend it.
• Reading by Will Wheaton
  Just kidding
• Beginning PyGame Programming
  This tutorial by Craig Maloney was my first real introduction to PyGame. Craig had a nice little demo environment all set up and ready for the presentation. He flew pretty quickly through many of the concepts behind PyGame while writing a Pong demo. Although he went through the material pretty quickly, I’m very interested in learning more about the platform as a result, so it’s safe to say the tutorial was a success as far as I’m concerned.
• Open Hardware with Arduino
  The speaker for this talk was W Craig Trader, and I have to say Craig was not only extremely knowledgeable about the Arduino (and obviously several other) platform, but also pretty excited about it, and since microcontrolers isn’t something I’ve ever bothered to learn anything about, I really didn’t expect to get so excited about the talk. The Arduino platform appears to be very accessible technically and financially, and also pretty powerful. Craig did an amazing job showing us the strengths and weaknesses of the platform and getting our whole group pretty excited to play with it!
• High Performance PHP
  This talk held by Rasmus Lerdorf, the creator of PHP and an infrastructure architect at Yahoo! Inc., took us through several performance optimization techniques for PHP apps, although many of the concepts featured in the talk could easily be applied to any apache-based app. It was very refreshing to finally see someone as experienced and well-rounded as him go through the tribulations of identifying and addressing performance bottlenecks in PHP apps. I was very interested in both the individual techniques highlighted during the talk, as well as the problem-solving process of a highly experienced software engineer, so that talk was the highlight of the con for me.

Unfortunately, due to very unfortunate logistical shortcomings, our group was unable to attend a lot of the panels we were looking forward to check out, but despite this, the con was a resounding success in every aspect you can think of. I’ve met some really cool and interesting people, learned a lot of very exciting stuff that will be guiding some of my personal research for months to come, learned some technical concepts that will directly impact my work performance, and had way too much fun.

The next stop on our list will probably be PyOhio. We had a chance to chat with Catherine Devlin, a fellow Pythonista & Oracle geek from IntelliTech Systems, who told us about it, and Aaron and I are looking into putting together a talk proposal for the con, if we can come up with it before the deadline, and we already have some pretty interesting ideas, so I’m looking forward to it. I’d also really love to attend PyCon 2010 in Atlanta!

If you’re interested in attending a convention where Linux and FOSS enthusiasts get a chance to have fun with the Sci-Fi crowd for a weekend of fun, PenguiCon is definitely for you!!

Share

No related posts.

CREATE PROCEDURAL LANGUAGE plpythonu;

Once the language bindings have been activated, you can start writing your stored procedures in python, however you should really read up on the following subjects first:

• CREATE FUNCTION statement syntax
• PL/Python Implementation Documentation

As an example, I’ve written a little SP in PL/Python to provide support for PCRE since the stock distribution of PostgreSQL only supports LIKE/SIMILAR, and POSIX Style Regular Expressions.

Let’s create our Python language binding, and create a standard text storage table:

-- Activate PL/Python
CREATE PROCEDURAL LANGUAGE plpythonu;
 
-- Create a plain text-storage table
CREATE TABLE text_storage
(
  id serial NOT NULL,
  payload character varying(128),
  CONSTRAINT text_storage_pkey PRIMARY KEY (id)
)
WITH (OIDS=FALSE);
ALTER TABLE text_storage OWNER TO xavier;
 
-- Let's throw in an index on the payload field for good measure
CREATE INDEX txt_payload_idx
  ON text_storage
  USING btree
  (payload);

Let’s now populate our table with some junk data:

INSERT INTO text_storage (payload) VALUES ('hello, world');
INSERT INTO text_storage (payload) VALUES ('the quick brown fox, blah blah blah');
INSERT INTO text_storage (payload) VALUES ('PCREs in Postgres');
INSERT INTO text_storage (payload) VALUES ('All hail Python!');
INSERT INTO text_storage (payload) VALUES ('Hello, test data!');
INSERT INTO text_storage (payload) VALUES ('Python would like to say Hello!');

And now we can go ahead and create our Python SP itself:

CREATE OR REPLACE FUNCTION pcre(text, text)
  RETURNS INTEGER AS
$BODY$import re
 
regex  = args[0]
in_str = args[1]
 
compiled = re.compile(regex)
 
IF compiled.search(in_str):
	RETURN 1
ELSE:
	RETURN 0$BODY$
  LANGUAGE 'plpythonu' VOLATILE
  COST 100;

As you can see, our PCRE matching system is extremely simple, yet pretty powerful. We import Python’s built-in re module, compile the specified regex argument, then attempt to match it against the other argument. Here’s a usage example on our test table:

SELECT id, payload FROM text_storage WHERE pcre('[H|h]ello', payload) = 1;
 id |             payload
----+---------------------------------
  1 | hello, world
  5 | Hello, test DATA!
  6 | Python would LIKE TO say Hello!
(3 rows)

As always, feel free to suggest any improvements.

Share

Related posts:

1. Fixing custom sequences in PostgreSQL
2. Ahh those cool little CLI tools…
3. Apache2 shortcuts

Inotail:

Inotail uses the Linux kernel’s inotify API, which was implemented with v2.6.13 to monitor changes to files on the filesystem. This design is more efficient than our beloved tail, which relies on polling the monitored file for changes every second. Example: To monitor in real-time syslog entries, try:

inotail -f /var/log/messages

The documentation for inotail, if you need it, can be found at http://distanz.ch/inotail/

Incron:

Incron is an event-scheduler similar to cron, except that it is based on file-system events as opposed to our beloved time-based cron daemons. It is also based on the inotify subsystem, which means it is only available on Linux as far as I know. Let’s set up a quick example to demonstrate the stuff you can do with incron. We’re going to install incron, and configure it to automatically create a thumbnail of any picture dropped in a specified directory using ImageMagick’s convert utility, on a stock Ubuntu Linux system:

# Package installation
aptitude install incron imagemagick
# Add your user account to the list of allowed incron users (replace xavier by your account)
sudo sh -c "echo xavier >> /etc/incron.allow"
# Create our directory structure
mkdir -p /home/xavier/images/original
mkdir /home/xavier/images/thumb
# Edit the actual incron file
incrontab --edit

The editor will now fire up. Enter the following lines in the editor, and exit:

# Convert /home/xavier/images/original/test.png to /home/xavier/images/thumb/test.png
/home/xavier/images/original/   IN_CLOSE_WRITE    convert -thumbnail 320x320 $@/$# $@/../thumb/$#
# When an original is deleted, automatically clean up the associated thumbnail
/home/xavier/images/original/   IN_DELETE    rm -rf $@/../thumb/$#

We’re all done. Any new image dropped in /home/xavier/images/original/ will automatically be converted into a thumbnail of the same name in /home/xavier/images/thumb/. There are many things you can do with incron, so i suggest you check out the following links:

• Incron Documentation: http://inotify.aiken.cz/?section=incron&page=doc&lang=en
• Incron FAQ: http://inotify.aiken.cz/?section=incron&page=faq&lang=en
• inotify.h: http://www.kernel.org/pub/linux/kernel/people/rml/inotify/headers/inotify.h

ccze

ccze is simply a logfile syntax highlighter for various file-formats commonly found on unix systems, such as syslog, apache logs, dmesg, etc… You can have it syntax-highlight a file in your terminal by using the following syntax:

ccze < /var/log/messages

Or you can pipe anything onto ccze to have it stream syntax-coloured output on your terminal. For example:

inotail -f /var/log/messages | ccze -A

Additionally, ccze can also output syntax-coloured text in HTML. For example, the following command:

dmesg | grep -i cpu | ccze -m html

Woud output the following document: ccze Output

Share

Related posts:

1. Extending PostgreSQL with Python
2. Apache2 shortcuts