Posts

Biari Workbench Technology Stack

Over the course of developing our Workbench solution we’ve adopted a powerful set of interconnecting components. It’s worth mentioning what these are and how they fit together.

Almost all the components of the stack are free and/or open source. We want to be as platform independent as possible and not get too locked in to one technology paradigm. This means that as much as possible, parts should be as “hot swappable” as possible – which also helps encourage strong componentisation. Using components with mature and open/standardised interfaces is very necessary when you’re crossing language boundaries (most notably, Javascript-Python and Python-C++) and client/server boundaries; otherwise you risk re-inventing the wheel. Ideally each component we use should also still be in active development (in the IT world – with the odd highly venerable exception – if software is not growing and evolving, it’s usually either dying, already in it’s death throes, or extinct).

There’s an art to using the right tool for the job, and we’ve made mistakes. We over-used Mako (see Loki’s blog post) and also originally used a slightly inferior lib for the C++ xmlrpc back end; both these mis-steps were fairly easily rectified. Arguably, we probably still use too much C++ and not enough Python – the C++ line count dwarfs the Python line count by a considerable margin. One last interesting point is that, at the moment, we’re still eschewing use of an ORM (Object Relational Mapping layer – such as SQL Alchemy) – time will tell whether that is a good idea or not.

Client:

JavaScript
– client-side browser language

jQuery
– JavaScript library for event handling and more

Cloudmade
– OSM map provider/server

 

Data Interchange:

JSON – JavaScript Object Notation

XML – eXtensible Markup Language

Mathematical Engines:

Mostly in C++ using STL

CppUnit
– C++ library for unit testing

OGR
– map data library, part of GDAL – used to read map data

Libxmlrpc-c
– C++ back end for XMLRPC – used by a running process to communicate with the front end via Python

 

Server:

Python
– language

CherryPy
– Python-based web app framework

PostgreSQL
– open source RDBMS

PsycoPg2
– database adaptor for PostgreSQL/Python

XmlRPCLib
– XML RPC (used to communicate with some of the engines)

Mako
– Python template library

Repoze
– Zope/WSGI Python middleware (for authentication)

 

 

Choosing a web framework

In the ongoing evolution of the workbench I’ve solved some interesting problems and found some hard ones I haven’t solved yet. I also made some poor choices and later found more elegant solutions. All this is a learning process and I’d like to share some of those findings in this series. I’ll cover templates, web framework choices and making your own widgets.

Templates:

Originally I searched for the most elegant and expressive templating language for what I needed to do and found Mako. It’s fast, good for simple interpolation and some basic control structures. However, after trying to write even one of our basic workflows in it, I was less enthused. I needed a range of slightly dodgy hacks to get what I wanted out.

For instance, I wanted to have a mako function put a piece of text (javascript) in a buffer and output it later (in the head section). I found a way of doing this:


<%!
    def string_buffer(fn, target_buffer='html_buf'):
        def decorate(context, *args, **kw):
            context['attributes'][target_buffer]  = runtime.capture(context, fn, *args, **kw)
            return ''        
        return decorate
%>

<%def name="doc_ready_js()" decorator="partial(string_buffer, target_buffer='doc_ready_buf')">
    ${caller.body()}

<%def name="main_js()" decorator="partial(string_buffer, target_buffer='main_js_buf')">
    ${caller.body()}

Needing to use decorators, partial function application and weird context/global variable magic for quite simple features freaked me out enough and convinced me to hunt for a better solution. I wanted the simple to be easy and the hard to be possible. The template was making the the simple hard. I looked at a pile of templating languages and I still liked Mako the most out of them, but decided after reading this wonderful rant by by Tavis Rudd that the solution is obvious: Do simple stuff with templates if you need to, but mostly avoid them.

The clearest way to demonstrate the advantages of a pure python solution is with code, both sections of code generate the same simplified workflow:

The Mako:


<%namespace name="zui" file="zui.mako"/>
<%namespace name="importer" file="importer.mako"/>
<%namespace name="widgets" file="widgets.mako"/>

<%inherit file="workflow_wrapper.mako"/>
<%def name="workflow()">
    <%
        current_workflow_id = attributes['get_new_id']()
        attributes['workflow_steps'][current_workflow_id] = [[],[],[]]
        grid = attributes['get_new_id']()
        map = attributes['get_new_id']()
    %>

    <%call expr="importer.importer({'name':'text', 'address':'text'}, ['name', 'address'], r'''
    Here you need to select a CSV file with address and a name for the locations
    you want to geocode. Names have to be unique.''', table_name,  'import_done()')">

    ## events are run on the client side so are javascript
    <%call expr="widgets.run_engine_step('Geocode',
                                         {'onclick':'''calculate_button('geocode_addresses',
                                         geocode_results);'''},
                                         workflow_icon='geocode')">

    <%call expr="zui.workflow_step('output', 'View Results')">
        ${widgets.grid_holder(table_name, grid)}
        ${widgets.map_holder(map)}
    

    <%zui:main_js>
        function import_done(){
            console.log("run when import is finished");
        }

        function geocode_results(){
            ${widgets.grid_init(grid, table_name, '')}
        }
    

The Python:


import zui, widgets, importer, layout

class Geocoder(zui.Workflow):
    def __init__(self, the_zui, table):
        zui.Workflow.__init__(self, the_zui, table)
        importer.Importer(self, {'name':'text', 'address':'text'}, ['name', 'address'], r'''
            Here you need to select a CSV file with address and a name for the locations
            you want to geocode. Names have to be unique.''', self.table_name,
            'Geocoder.import_done()')

        ## events are run on the client side so are javascript
        widgets.RunEngineStep(self, 'Geocode',
                {'onclick':'''calculate_button('geocode_addresses',
                    Geocoder.geocode_results);'''}, workflow_icon='geocode')

        grid = widgets.Grid(self.table_name)
        cm_map = widgets.Map()

        self.workflow_step('output', 'View Results', container_size = "container-results1", body =
                layout.multi_column('', grid, cm_map))

        self.main_js(
            '''
            function import_done(){
                console.log("run when import is finished");
            }
            function geocode_results(){
                '''+grid.load()+'''
            }
            ''')

It is so much easier to be expressive and elegant when you minimise the usage of templates. You can clean things up by using some of the expressive power of a multi-paradigm language and not have to do anything too strange. I still see the use for templates when you need to interpolate a few variables into a long piece of HTML. The built in python template function seems fine for this. In my next post I’ll over web frameworks and what I think about widgets. You should be able to see from today’s bit of code that I think they are important and hard to do with templates.

Loki

p.s.

If you feel like relaxing after reading this, check out my band’s YouTube channel. 🙂

Thoughts on Point Solutions

Lately I have been thinking a bit about the advantages of small, tightly focussed web apps (so-called “point solutions”) that scratch a single little itch, versus larger, more powerful and general web apps that tend to deliver more of a total body rub. This question is of utmost importance to a company like Biarri that needs to place its development time and effort into the best channels.

The question was highlighted by a real-world problem a colleague posed recently: how to assign foursomes in rounds of Golf so that all of the players got to play with each other player at least once. It is not trivial to construct such a solution (if one even exists) by hand, if the constraints are “tight” enough (for example, 20 players and 8 rounds).

Small point solutions that solve a small but non-trivial problem like this might be fairly quick to develop and deploy on the web. But it doesn’t take much feature creep before you get a pile of extra “features” (particular requirements for some players, minimising the number of repeated pairings, right through to printing out score cards etc); before you know it (or more precisely, after months or years of hard coding) you’d have a full-blown Golf Tournament Scheduler. Such a web app might sell for much more, but would probably attract many less customers. And what happened to the poor casual golfer or golf tournament organiser on a shoestring budget who just wanted to solve his or her original golf player assignment problem?

In the spirit of acknowledging that the future is impossible to predict, I think Biarri must address more wide-ranging, lightweight “point solutions”, particularly at our fledgling stage. More mini-apps with a wider potential customer base will allow us to gauge which itches need the most scratching; more complex apps, as every seasoned developer knows, seem to always cause issues and problems – in short, sheer complexity – quite out of scale with the larger code line count; not to mention being harder to use and understand for users (more buttons!)

Those who have test-driven our Workbench solution will also know that, to some extent, we’re trying to have our cake and eat it to, by allowing these smaller “point” solutions to exist as workflows (standalone web apps) in their own right, whilst also being “nestable” – that is, able to be composited in a larger, more powerful workflow. Look out for Geocoding as a sub-workflow inside Travel Time Calculation, coming to the Workbench very soon. And who knows if the Biarri Golf Tournament Organiser will ever eventuate!

Generating KML Thematic Maps


The Biarri workbench now has a tool for showing the frequency of an event per postcode. The colourised map (funkier name Choropleth map) is generated as KML from postcode regions that are stored in a postgis database. This just requires a little bit of sql and python:


dict_cur.execute("select p.gid, p.poa_2006, ST_AsKML(the_geom) as \
                  polygon, frequency, (frequency - stats.min) / (stats.max - \
                  stats.min)::float as scaled_frequency from "+table+" c, \
                  postcode_regions p, (select max(frequency), min(frequency) \
                  from "+table+") as stats where p.poa_2006 = c.postcode")
return my_lookup.get_template("thematic_kml.mako").render(rows=dict_cur.fetchall())

The sql just normalises the column we want as our colour intensity and returns the KML for each postcode region.

The mako template:




    
    Thematic Map
        generated with the Biarri workbench
    % for row in rows:

 
 ${row['poa_2006']}
 
 ${row['frequency']}
 
 #${row['gid']}
 ${row['polygon'].replace(" ", '\n')}
 
 % endfor
 

<%!
import webcolors
import colorsys

def intensity(frequency):
#frequencies are already scaled between 0 and 1
 return 'cc' webcolors.rgb_to_hex(tuple((int(a * 256) for a in colorsys.hsv_to_rgb(0.134, frequency, 0.90)))).strip('#')
%>

The mako template is quite easy to read if you notice the % sections are actually python code. The only challenge I had was that ST_AsKML produces kml with the co-ordinates space separated. Google earth is fine with this but the kml viewer we’re using, web map lite, only likes them newline separated. My colour function just changes the intensity of the colour. A gradient between blue and red would probably be more appropriate.

Loki

The benefits of hosted solutions

Cloud computing offers many advantages over more traditional software channels. There is an interesting analogy here between computing power and electricity (elucidated well in the book The Big Switch: rewiring the world, from Edison to Google by Nicholas Carr 2008) – cloud computing offers computing power and business applications served up from a “centralised power plant”.

At Biarri we are excited about the advantages of cloud-hosted solutions instead of the more traditional desktop approach, in particular:

  • All clients run the latest and greatest version of the software
  • Accessible anywhere there is an internet connection
  • From a development point of view, problems can be solved once in a controlled computing environment. This lets developers concentrate more on features of value to customers and less on making software robust enough to run under many different environments (OS versions or variants for example).
  • Avoiding desktop licensing issues (e.g. tying software to a particular machine with awkward process for transferring license to another machine)
  • Short-circuits problems with desktop software deployment
  • Instant support – pursuing the electricity analogy, this would be like having a handyman always immediately on call to fix your problem
  • Scalability – ability to increase the computing power required on demand by “spinning up” more machine instances in the cloud
  • And best of all for our business clients – available on a low-cost monthly subscription (with the ability to easily turn on and off the service) without an arduous IT deployment

Biarri’s Workbench continues to be developed and enhanced and offers a hosted solution that comprises many different optimisation engines that all “plug in” to a central, flexible hosted architecture. In keeping with Biarri’s key values of accessibility and power, the user interface offers both a workflow-oriented interface with rich data manipulation and visualisation functionality.

Accessibility – at work every day

The posts below describe Biarri’s approach to deliver optimisation through accessibility of a number of different channels. Whether it is modelling in excel, a Biarri inside engine, by using the Biarri Workbench or even a managed service – these are all great ways for businesses to quickly and affordably benefit from the power of optimisation and Operations Research.

On a more practical level each day we are seeing examples where Biarri’s open and accessible approach is giving customers the power to “drive” the model or application and get the benefits of new insights and approaches to complex problems which deliver savings.

The power of being able to change parameters and input data and re-run optimisations all in a matter of seconds means in a very short time customers have used Biarri’s tools to get a deep understanding of what drives cost and yield in a supply chain or business operation and the impact of the different levers they can pull to affect change.

We at Biarri believe that powerful insight and analysis should be available quickly, to answer the burning questions that management have. For this reason, every day the team at Biarri look for new ways to make our commercial mathematics more accessible for our customers.

Accessible Engines: Biarri Inside

From the start Biarri has been committed to providing easy access to powerful optimisation.  Our focus at inception was twofold being:

  • Custom optimisation models delivered through consultancy projects and
  • To deliver task specific engines made available on a Software as a Service (SaaS) basis via the “Biarri WorkBench” (WB).

The strong demand for our analytical consulting services continues to surprise us and the WB is now in Beta with a launch expected later in the year.

However, a new delivery channel has emerged to provide even greater access to our optimisation capability.  Over the last few months we have been developing optimisation formulations and engines that other companies are embedding in their own software.  This ‘Biarri Inside’ approach allows companies to quickly and affordably add optimisation capability to an existing application and leverage Biarri’s expertise in formulating and coding optimisation engines.

We are responding to the demand for the ‘Biarri Inside’ approach by ensuring the optimisation engines that we develop have a well documented and designed API so  each can be easily incorporated into existing solutions.  We are now involved in a number of projects that leverage this delivery channel for accessible optimisation.