Benjamin Thomas

On the Interwebs, number 20

2010-01-11T00:00:00+00:00

Fun with Python, OpenCV and face detection — I have been looking for an article on this for a while. This is so amazingly cool!

Reading the Audio Stream from Firefox 3.7 — I hope all the browsers get on this. This would be absolutely awesome.

ECMAScript 5’s Date Extensions — Some long time coming changes to the date object in Javascript (aka ECMAScript). The most welcome of which are the new ways to instantiate a Date object.

ECMA 5/Mozilla Features Implemented in V8 — Speaking of ECMAScript 5 features, a list of which ones are supported by V8, the javascript engine behind Google Chrome and Node.

How Do You Look When Merging Fails ;-) — A script that takes a picture with your webcam every time your merge fails. Pretty funny idea. Though, Mercurial only as far as I can tell.

Asynchronous Servers in Python — A list of all the async web servers available in Python with example code. It is interesting to see the different approaches to one problem.

arel — A way to write SQL in Ruby without having to worry about different SQL engines. I was going to try and write something like this, so it saved me a lot of trouble!

Rails and Merb Merge: Plugin API (Part 3 of 6) — With this merge with Merb, Rails is taking the “framework wars” up a notch. The stuff they are doing is really cool. Also checkout ActiveModel and Generic Actions.

On the Interwebs, number 19

2010-01-06T00:00:00+00:00

Unicode Code Converter — Displays the given text in many different escaped formats like HTML, URLs, Javascript, etc. Sounds odd at first, but just try the example and it will instantly make sense.

Clarity — Looks like a great tool for monitoring log files remotely.

File APIs in Firefox 3.6 — hacks.mozilla.org has a great series of articles on using the new File APIs in the upcoming Firefox 3.6.

Real time online activity monitor example with node.js and WebSocket — Websockets look really powerful. And we already know I am obsessed with Node.

Firebug and the JIT — Even if you don’t have Firebug enabled it is probably slowing down your browsing experience.

Node.js on IRC: — A post with a collection of links on using Node with IRC.

On the Interwebs, number 18

2009-12-19T00:00:00+00:00

A piece with a lot of screenshots about the close tab behaviour in Google Chrome — Excellent write up about some excellent design. There has been a bug filed for Chrome to change this behavior on Mac OS X so that the close button goes on the left. I think this is lunacy, the Chrome guys did a great job here.

CSS Backgrounds in Firefox 3.6 — Finally we are getting the ability to manipulate CSS background images. Though we still don’t have the one feature I want: the ability to specify positions relative to a given point, like “put this image 6px to the right of center”.

Using YQL to load and convert RSS feeds really, really fast — I really need to get around to wrapping my head around YQL. I just wish it was more distributed so you aren’t relaying entirely on Yahoo.

RightJS 1.5: 6-8 times faster than jQuery — Some excellent analysis of performance claims. The moral? Don’t trust everything you read without looking at the details!

CSSEmbed — Convert images referenced in a CSS file into data URIs. Pretty cool!

Looking for tennis courts on aerial photos — I wish I was better at Computer Vision techniques. But this is a nice explanation!

On the Interwebs, number 17

2009-12-18T00:00:00+00:00

A JPEG Encoder for JavaScript — Just what it says. Pretty amazing feat actually. Javascript is really coming along these days!

A short PayPal horror story — I have not had a good experience with using PayPal as a vendor either. From this day forth, when presented with an option on checking out I am not longer going to choose PayPal.

24 Ways — 24 Ways is back this year, and has some good stuff. 24 Ways is a CSS advent calendar with 24 articles on CSS, one a day until Christmas.

Performance Advent Calendar 2009 — Inspired by 24 Ways, but about performance.

Ditz — Issue tracker intended to be embeded in your source. I love this idea.

ImageOptim — Just what it says: “ImageOptim optimizes images — so they take up less disk space and load faster.” Dead simple and works well.

JavaScript Charts Roundup — Summarizes different javascript libraries for graphing. I’ll be looking at this for a project I started a little while ago.

• Bomber – a Node web framework — Actions

2009-11-29T00:00:00+00:00

This is another post in my series of posts about designing a web framework using Node. Catch the first two here and here.

A New Website

First a quick update: last night I decided Bomber needed its own place on the web, so I threw together a website for it. It is mostly based on the code for this site, so it was pretty quick. Anyway…

Announcing bomber.obtdev.com! I’m pretty proud of the icon, so make sure to check it out. Now I have a place to put documentation and more details then are appropriate for blog posts.

Actions

Today’s installment is going to be about what I am calling “actions”, by which I mean, the code that is run for each request. The last post talked about routing, which was the business of figuring out from the URL which code should be run. So, this is the second half of that equation. For comparison, Rails calls these “actions” and Django calls them “views”.

Conceptually actions are pretty simple. Given a request, it is their role to generate a response. If you requested an HTML list of blog posts, it is the job of the action to generate that HTML document. In an MVC framework, the action plays the role of the glue between all the different parts of application: the database, the templates, the cookie or session variables, etc. Here is an example of an action from Rails, generated by its scaffold command:

def index
  @photos = Photo.all

  respond_to do |format|
    format.html # index.html.erb
    format.xml  { render :xml => @photos }
  end
end

The code is pretty straightforward (even if you aren’t familiar with Rails). It loads a list of photos from the database, and then depending on the requested format it either renders the ‘index.html.erb’ template, or converts the list to an XML document.

Actions have a lot of different needs. The two most basic are dealing with the request and the response:

An action needs to have access to the details of the request: What HTTP method was it? What headers were sent? What were the query parameters? What was the body of the request? What POST variables were sent? What cookies were sent? What session is it associated with? What session variables are set? Where is the request from? What browser is the request from? And so on.
An action needs to be able to modify all the details of the response that it is creating. Primarily, that is setting headers, cookies, or session variables and generating the HTML.

Giving users access to the Response and the Request in existing frameworks

Django is very explicit, here’s an example of a simple view from the docs:

from django.http import HttpResponse
import datetime

def current_datetime(request):
    now = datetime.datetime.now()
    html = "<html><body>It is now %s.</body></html>" % now
    return HttpResponse(html)

You are given a request, and you return a response. This is very intuitive, and powerful but can be a bit of a pain sometimes when you don’t want to be writing all that boilerplate code over and over again. I wish I just didn’t have to type quite so much. (This is a complaint I keep coming back to over and over again with Django, and it really isn’t a strike against Django. Django is this way precisely because they don’t want to pigeon hole you into writing your projects in a specific way. But sometimes I just wish they would pick a way for me)

Rails is a bit more “magical” in the sense that a lot of the code happens behind the scenes. With Rails, an ActionController object is iniated for each request, and then the requested action method is called on that object. And because self is optional in Rails it seems like things just come out of nowhere. Here’s an example action:

def show
  @photo = Photo.find(params[:id])

  # just so you can see what it is like to set session and cookie vars
  log session[:key] # get the session variable from the response
  session[:key] = "value" # set the session variable for the response

  cookies[:key] = "value"

  # and setting headers
  response.headers["Content-Type"] = "application/pdf"

  respond_to do |format|
    format.html # show.html.erb
    format.xml  { render :xml => @photo }
  end
end

What is going on is params, session, cookies, response and respond_to are all actually methods on the ActionController object. To return a response you call a method like render, or redirect_to. I really do feel like writing applications in Rails is like writing in a-whole-nother language. Rails has request and response objects, but you generally don’t use them in favor of the other simpler methods. With Rails there isn’t a very clear distinction between requests and responses like there is in Django.

Sinatra works in a very similar way to Rails. And of the Node frameworks, the majority of them are Sinatra like in spirit.

Evented programming

If we already didn’t have enough choices for how to design an API for a web framework, Node then adds an additional level of complexity due to its evented nature.

What does that mean? Well, Node lives by the philosophy that a program should never just sit there and do nothing while it is waiting for input or output to finish. Here’s an example of what not to do from Ryan’s presentation at JSConf this year:

var result = db.query("SELECT..");

Ryan asks a simple question, “What is the software doing while it queries the database?” Well, with most libraries it is doing nothing, it is just sitting there waiting for the database to do its thing. This is inefficient. An evented library uses callbacks, so instead of waiting for the database to return, it does some other computation, and then when the database query is done, it calls the callback function. Here’s Ryan’s example:

db.query("SELECT..", function (result) { 
  // use result
});

This is pretty straight forward but what if you need to make a second database call depending on the result of the first one, and then use both those results? You’d get something like this:

db.query("SELECT..", function (result1) { 
  db.query("SELECT..", function (result2) {
    // use both results here
  });
});

And what if you need to load in a template file (which would mean reading the file from disk) and use that template to format your results?

db.query("SELECT..", function (result1) { 
  db.query("SELECT..", function (result2) {
    templates.load("file-name", function(template) {
      // use both results and the template here
    });
  });
});

There is an additional problem here. How do you keep track of state between all these different functions? In the previous example it is easy because a function is aware of all the local variables that exist at the time it is created, so the last function has access to result1 and result2 but this isn’t always the case. Especially if you are trying to reuse functions in different parts of the app.

This is very quickly getting unwieldy. And thus far none of the Node frameworks have addressed this issue head on. There have been some talks on the mailing list of adding deferred objects to Node and I think many people hope that this is going to solve the problem of writing evented web frameworks. While I think deferreds would be awesome, I think the solution needs to be a little more integrated then that.

Bomber Actions

(Note, this is the API I plan to implement in Bomber, but not all of it is completely written yet. Some of it though!)

In Bomber, the solution is Action objects, which have deferred functionality, but are a little more aware of the problem they are trying to address. Specifically, an Action has a list of tasks that should be run to complete the action. And just like with deferreds, the result of a previous task is passed to the next task. This way it is easy chain functions to be run after one another. However, (and this is the real bread and butter of Actions) unlike with deferreds (at least as far as I can tell) if the return result of a previous task is itself a deferred (or a Promise in Node-speak), the Action will wait for that deferred to finish its task before calling the next task. Now you can chain up all your tasks, and not worry about the asynchronicity of the different parts.

Action objects have a couple other tricks up their sleeves.

They will make sure that all tasks are bound to the action object itself, so you can keep track of state between tasks by setting variables on this.
Every task will be called with the request and the response for that Action as the first two arguments. This makes it easy to access those objects so you can get/set cookies, sessions or headers.
If any task returns a certain kind of object (like a ‘404 error’ object – I haven’t decided what to call it, yet) Node will stop running the tasks for that Action and send that as the response for this request.
If the result of the last task is a string or an object, Node assumes you want to use this value for the body of the request. If it is a string it assumes it is HTML and if it is any thing other than either a string or one of the objects in the previous bullet point, it converts it to JSON and sends that. This allows you to easily construct simple responses, but if you want more control you can just return null and manage it yourself.

Here is how the Server initializes an Action:

var action = new Action(request, response, view_function);
action.start();

Then a possible view_function could look like this (to tackle the example from the section on evented programming):

function view_function(request, response) {
  // in this case the db.query function is assumed to return a deferred object
  this.addTask(function(request, response) {
    return db.query("SELECT...")); 
  });

  // the next task is called with the result from the previous deferred object
  this.addTask(function(request, response, result1) {
    this.result1 = result1;
    return db.query("SELECT...")); 
  });

  this.addTask(function(request, response, result2) {
    this.result2 = result2;
    // I'm assuming templates.load is a deferred object as well...
    return templates.load('file-name');
  });

  this.addTask(function(request, response, template) {
    // use both results and the template here
  });
}

Now, I admit that at this point this looks like it might have been more work. But I guess I am going to have to try it out some to really see. I think this potentially could allow for better use of DRY programming because now it is trivial to use predeclared javascript functions as opposed to anonymous ones. That with the additional benefit of being able to return 404 errors or 3xx redirects at any point in that process will make this pretty handy.

I have one more post planned about this Bomber design stuff but it might be a little while before I get to it. (I need to figure out how I want templating to work before I can write it! But I have some ideas…)

On the Interwebs, number 16

2009-11-28T00:00:00+00:00

Some websites I have found interesting recently (in no particular order):

(Whew! This is a long one! Apparently I waited a little too long to post these!)

Sammy — A Sinatra inspired framework for making browser based javascript applications. This is seriously cool. (Don’t confuse this with server side javascript applications. This is for writing an application that lives entirely in a browser.)

Mail — What appears to be a spectacular Ruby library for checking/sending/generating/parsing emails.

Identity in the Browser — A very cool mockup of what it would look like if logging-in and managing accounts on websites was built into the browser. This feels like a hint of the future. And I can’t wait! Putting the controls in the URL bar makes perfect sense.

View Cookbook for SQL Jockeys — An explanation of CouchDB views for SQL people. Honestly, while this was helpful, I need a lot more examples before I am going to truely get this map/reduce stuff for CouchDB.

Using files from web applications — Mozilla’s documentation on a new javascript feature that allows you to access files without having to upload them to a server first. Bit by bit browsers are aquiring the tools to completely replace native apps. I can’t wait!

LABjs: new hotness for script loading — A new library for getting your javascript files to download more quickly. I take a different approach (which is to make sure I am only using 1 javascript file) but this looks pretty useful for larger applications that have lots of dependencies.

An interesting 404 error — Seems like it would only be interesting to people who are familiar with HTTP status codes, though.

And now some Node.js links (I’m grouping them here in case you aren’t interested in my latest obsession):

Video: Node.js by Ryan Dahl: Video of the talk whose slides I linked to last week. The talk honestly doesn’t add that much to the slides (he did some good work with those!) but it is really fun to put a face to the person.

restler — A super simple library for making HTTP requests from Node. I don’t need this currently, but this looks like everything I would want.

• Bomber – node.js web framework — URL Routing

2009-11-24T00:00:00+00:00

This weekend I finally got a chance to play around a little bit with some of the ideas I talked about in my last post on writing a new web framework with node.js.

Introducing Bomber

I have decided to call the framework “Bomber”, which is a variant spelling of “Bomer”, which is a combination of “Ben” and “Omer”, which is because my friend Omer has been volunteering API/design thoughts and advice.

You can checkout the source code for Bomber from GitHub if you want to see how it works or try it out. You must have node.js installed¹ first.

Bomber is centered around the idea of “apps” (much like Django is), where an app is just a way to group code together in a folder so that it is more portable/reusable. Unlike with Django, Bomber expects certain code to live in certain places. For example, the URL routing has to be in a file called routes.js. I have written up a description of what files an app can have in the docs folder of the source.

To actually run the Bomber server, cd into the source code directory and run the command ./bomber.js ./exampleProject. This will start the server, using the ‘exampleProject’ app.

Right now Bomber doesn’t do much. If you are looking for something more stable may I suggest one of the more established projects?

Bomber Routing

I’ll show you how it works first, and then after, I’ll talk about why I decided to have it work the way it does.

The goal of the routing in Bomber is to determine which function should be run given a URL.

Here is an example of what a routes.js file could look like:

var Router = require('bomber/lib/router').Router;

var r = new Router();

r.add('/', { view: 'simple', action: 'index' });
r.add('/section', { view: 'simple', action: 'section' });
r.add('/section/:id', { view: 'simple', action: 'show' });
r.add('/:view/:action');
r.add('/other_app/:action', { view: 'view_name' });

exports.router = r;

All this file needs to do, is create a Router object, add routes to it, and then finally export that router object.

If you have used Rails this should look very familiar.

How adding the routes works is you give the router the URL to match, and then an object with the set of parameters that should be used to decide which function to run.

The app parameter tell Bomber which app to use, view tells it what view file to look in, and action tells it the name of the function to run. Any other parameters (or those declared in the URL, like :id) will find their way to the function (I’m not sure how yet, wait for another post!). If you don’t specify app, Bomber will use the current app.

Additionally, if you want complete control over the process, you can give it a regular expression instead of a string for the URL to match. Behind the scenes all routes are converted to regular expressions, anyway.

The Thought Process

There were 3 ways I considered using when designing this: the Sinatra way, the Django way, and the Rails Way.

The Sinatra way is very popular with node.js frameworks (I have yet to find one that doesn’t do it this way). The Sinatra way is to have the routing in the same file and even inline with the code that is to be run. It looks something like this (to take an example from Express):

get('user/:name/:operation', function(){
  param('operation') + 'ing ' + param('name')
})

Or like this (to take an example from Djangode):

dj.makeApp([
  ['^/$', function(req, res) {
    dj.respond(res, '<h1>Homepage</h1>');
  }],
  ['^/other$', function(req, res) {
    dj.respond(res, '<h1>Other page</h1>');
  }],
  ['^/page/(\\d+)$', function(req, res, page) {
    dj.respond(res, '<h1>Page ' + page + '</h1>');
  }]
]);

This is a great system, as long as a) you don’t have too many routes and b) your functions aren’t too large. Otherwise I feel like it gets a bit unwieldy trying to keep track of what happens where. I’m kind of envisioning Bomber being used for slightly larger projects. And in that case it is nice being able to see all your routes at one glance.

This means I’d like the routes to be in a separate file from the functions.

The Django way works like this. You have a file in which you declare your routes, and each route maps to a specific function else where in the app. Here’s an example from the docs:

urlpatterns = patterns('',
    (r'^articles/2003/$', 'news.views.special_case_2003'),
    (r'^articles/(\d{4})/$', 'news.views.year_archive'),
    (r'^articles/(\d{4})/(\d{2})/$', 'news.views.month_archive'),
    (r'^articles/(\d{4})/(\d{2})/(\d+)/$', 'news.views.article_detail'),
)

I have two complaints about the way Django does routing.

I have to write out a route for every different function I want to use.
Named groups aren’t very elegant (and this is Python’s fault not Django’s).

But there are two things I really like about the way Python does routing.

It is great being able to just write a regular expression. I like that power.
You can pass off the processing of a URL to other routing files. Here’s the example from the documentation:

urlpatterns = patterns('',
    (r'^weblog/',        include('django_website.apps.blog.urls.blog')),
    (r'^documentation/', include('django_website.apps.docs.urls.docs')),
    (r'^comments/',      include('django.contrib.comments.urls')),
)

This is really great. And is something I hope to add to the routing for Bomber soon.

The Rails way works very similar to how I ended up doing routing in Bomber.

It is also very similar to how Django does its routing. There is a subtle difference, however. With Rails, instead of having a route match to a function, a route just generates an object which has all the parameters needed to find the function. Then Rails uses this object to actually do that. The difference is that you can use parameters gathered from the URL to tell Rails which (to use Rails lingo) controller to use, or which action to run. This means you can set up defaults and not have to do anything else (if you want to). Even if you add controllers or actions later, your defaults still handle everything. Here’s an example from the Rails Guide on Routing:

map.connect ':controller/:action/:id'
map.connect ':controller/:action/:id.:format'

This solves my first complain about Django.

I also like it much better how you don’t have to think about regular expressions if you don’t want to. “Give me everything between this slash and this one and call it X” seems so intuitive to me. This solves my second complaint about Django.

In the end I am hoping to have a system that takes the best parts from Django and Rails, and I think I am on my way.

Simon Willison has a great write up of how to get it installed (I can’t link directly to that section, scroll down). And there is a Mac OS X installer if that is easier for you. But the installer will soon become out of date. It will probably be easiest to stay current if you checkout and build from source. Node is changing quickly these days! ↩