Wednesday, June 29, 2005

PEP 238 - import .relative

Python Imports

PEP 238 which has been accepted and may have an impact on any Python packages you've made so far, depending on your import style. This PEP will change the way imports work in a package. By default, all imports will be concidered absolute. Previously, if you had a Python package named 'Foo' with modules 'Bar' and 'Fig', you could import 'Fig' from the 'Bar' module by typing 'import Fig'. Once this new feature is implemented in Python 2.5, it will look for 'Fig' in sys.path first. They also plan to implement relative imports as well. With this you will be able to import the 'Fig' module from 'Bar' by typing 'import .Fig' for more a better example, see Guido's Decision or read the entire PEP.

Other features include importing groups like:

from os import (path,
sys,
mkdir)

Which allows for easyier multi-line imports. Check out the PEP for additional information.

Note that you can use the new features in Python 2.4 using:

  'from __future__ import absolute_import'

In Python 2.5, any package that results in a relative import from a module from within a package will result in a DeprecationWarning. Python 2.6 will include the fully implemented and on by default PEP 238.

PEP 238 - Abstract

The import statement has two problems:

  • Long import statements can be difficult to write, requiring
    various contortions to fit Pythonic style guidelines.

  • Imports can be ambiguous in the face of packages; within a package,
    it's not clear whether import foo refers to a module within the
    package or some module outside the package. (More precisely, a local
    module or package can shadow another hanging directly off
    sys.path.)

For the first problem, it is proposed that parentheses be permitted to
enclose multiple names, thus allowing Python's standard mechanisms for
multi-line values to apply. For the second problem, it is proposed that
all import statements be absolute by default (searching sys.path
only) with special syntax (leading dots) for accessing package-relative
imports.

Feedback

Let me know if this post was useful or not, so I'll know wether or not to post something like this again. I hope everyone finds this post to be useful.

Saturday, June 25, 2005

Fourth SoCal Piggies meeting

The SoCal Piggies had their fourth meeting at USC on June 21st at 7:00 PM. Seven Piggies attended -- in order of appearance: Daniel Arbuckle, Howard Golden, Grig Gheorghiu, Diane Trout, Mark Kohler, Charlie Hornberger and George Bullis.

Diane presented a [WWW] tutorial on SimpleTAL, a Python-based templating language derived from Zope's ZPT. [WWW] SimpleTAL allows you to lay out dynamic content in HTML or XML form. Diane showed us examples of XHTML templates that contained SimpleTAL-specific expressions such as tal:content, tal:condition, tal:repeat. The nice thing about SimpleTAL templates is that they can be edited by Web designers using their HTML editing/authoring tool of choice, and the tal expressions will simply be ignored by the editor. This enables a clean separation between presentation and content and lets programmers concentrate on the business logic aspect of the application (written in Python of course :-). Diane also showed us a demo of a Quixote-based Web application called [WWW] CompClust that she developed at Caltech. CompClust is a bio-informatics app that uses various clustering algorithms for studying relationships between genes. The app also makes heavy use of [WWW] matplotlib to generate some very nifty graphics on the fly. The templating engine for the application is of course SimpleTAL.

The second speaker of the night was...Diane again. She presented a [WWW] tutorial on Pyrex, a language for writing Python extension modules. [WWW] Pyrex allows you to write code that mixes Python and C data types, and compiles it into C code as a Python extension module. Diane used Pyrex in her bio-informatics Web application to call C code that runs clustering algorithms from her Python code. One less known use of Pyrex is to compile Python programs into binaries that can then be used as stand-alone executables without the Python interpreter. George asked what does Pyrex do that SWIG doesn't, and Diane said Pyrex is more forgiving when it comes to interfacing with C code, and also allows you to more easily write Python code that interacts with the C function calls. For more details on Pyrex, see also this [WWW] Charming Python tutorial from IBM developerWorks.

[Note on viewing the S5-based slides that Diane put together: for the controls that allow you to move back and forth through the slides, move the mouse cursor to the middle bottom part of the screen]

We also had some lively discussions on ways to dynamically generate HTML and possibly reuse HTML snippets across an application, especially in the case when a template mechanism would be too static (a subject brought up by Mark). Several possibilities were mentioned: the htmlgen module, good ol' CGI, rolling your own acquisition mechanism similar to the one used in Zope, and of course the more heavy-weight approach of using a Web application such as Quixote. The subject of GUI widget toolkits was also brought up, especially ways to build GUIs in a cross-platform, cross-tool way by specifying for example the names and values of the widgets in an XML file. One could then ideally use any GUI toolkit in order to actually display the widgets on the screen. [WWW] XUL was mentioned as one technology to look at in this arena.

We also discussed a bit about unit testing, and I mentioned [WWW] py.test as an alternative to the standard unittest module. I think a presentation on Python's unit test frameworks would be interesting, I wrote some tutorials that I can put together as a slide show for a future meeting.

Many thanks to Diane for her presentations and to Daniel for hosting the meeting.

The next meeting will be at USC again, on a date to be announced. Topics for next meeting:

  • Demo of an interactive multi-player game written as a Python CGI application that uses AJAX techniques (Charlie)

  • Tutorial on the [WWW] py library (Grig)

Sunday, June 12, 2005

twill release

(Repost, via the socal-piggies blogger stuff:)

twill 0.7.1

Coooooome and get it! Announcement, docs, download.

This is basically a bug-fix version with about 15 different minor
fixes. Several users requested things like HTTP basic auth handling
and file uploads, and I patched several bugs in the underlying
packages (urllib etc.) Internal ugliness was fixed or augmented,
depending on whether or not I was dealing with forms, which is
inherently ugly code.

Once I've got all the desired features in there, I can start cleaning
up the internal code; right now I'm focused more on testing what's
there (which will make cleanup easier!) I've also gotta figure out
a way to do meaningful unit tests. That's for v0.8.

For the summer, I will be gone. I predict I will not be working on twill
until I get back in late August...

--titus

Saturday, June 11, 2005

Simple function optimizaton

This is an optimization technique I learned about a few days ago at a symposium. "Optimization" in this case means finding the input parameter values that result in the best function output, or at least approximating them.

The performance of this implementation isn't as good as the presenters', in terms of the number of iterations it takes to approach the optimum. I'm not sure what the cause of that is, but it may well just be a case of their having optimized their optimizer (two different meanings for the word "optimize" in the same sentence... there oughta be a law.)

In any case, this is a pretty nice algorithm, which converges relatively fast and (perhaps more importantly) is quite simple and understandable. Once you understand it, you'll be able to code it out from memory.

The general idea is that the optimizer maintains a "swarm" of "particles" which move about the space of the function, trying to find the best locations. The rules used to control their movements are (loosely) inspired by those that guide the flocking behavior of birds and fish.

This implementation is perhaps obfuscated by the use of numarray, but the advantages are too great to ignore.

from numarray import array, greater, maximum, minimum, transpose
from numarray.random_array.RandomArray2 import random as randarray

def pso(function, ranges, particle_count = 5, friction = 0.95, confine = True):
r"""
Performs a Particle Swarm Optimization on the passed function,
performing one optimization step for each iteration step. The
yielded values are are 2-tuples of (best known value, tuple of
parameters resulting in the best known value).

Parameters:

* function: The function you wish to find the optimal inputs
for. It should be a function of X floating point parameters,
where X = len(ranges). Further, the result should be a floating
point number which can be interpreted as the function quality
with a given set of parameters. If your desired function does
not result in a float or is not parameterized by floats, feel
free to wrap it in a function that does appropriate translation.

* ranges: The bounds within which the exploratory particles will
be created. If confine = True, these ranges also bound the space
to be explored. Each range should be a 2-tuple of (lower bound,
upper bound), and there should be one for each parameter of the
function.

* particle_count: The number of particles to maintain while
optimizing. Fairly small numbers are often good. Try 5 if you're
not sure.

* friction: The amount of 'velocity' that is lost from the system
on each time step. Friction will be clamped to the range [0-1],
and then mutuplied by the current particle velocities. A value
of 1 will tend to cause the optimizer to fail if the boundary
flag is not set to optimize.reflect. If you're not sure what value to
use, try 0.95

* confine: May be set to False, allowing exploration beyond the
bounds set by the ranges parameter.
"""


dimensions = len(ranges)
upper = array([r[1] for r in ranges])
lower = array([r[0] for r in ranges])
shape = (particle_count, dimensions)
positions = randarray(shape)
for pos, rng in zip(positions, ranges):
pos *= rng[1] - rng[0]
pos += rng[0]
velocities = randarray(shape) / randarray(shape)
lbest = array([function(*pos) for pos in positions])
lbestat = array(positions)

gbestat = lbest.argmax()
gbest = lbest[gbestat]
gbestat = array(positions[gbestat])

while True:
velocities *= friction
velocities += randarray(shape) * (array([gbestat]).repeat(particle_count) - positions)
velocities += randarray(shape) * (lbestat - positions)

positions += velocities

if confine:
positions = minimum(maximum(positions, lower), upper)

values = array([function(*pos) for pos in positions])
replace = greater(values, lbest)
lbest[replace] = values[replace]
lbestat[replace] = positions[replace]

mval = values.max()

if mval > gbest:
gbest = mval
gbestat = positions[values.argmax()]

yield gbest, gbestat