Wildcard Routing is an Anti-Pattern April 21, 2015

Probably the most cringy pattern I see when creating apps is wildcard routing. With the next version of Express changing support for wildcards, many people have already begun complaining.

const error = require('http-errors')

app.use('/thing/:id/*', function (req, res, next) {
  Thing.get(req.params.id).then(function (user) {
    if (!user) throw error(404)
    req.user = user
  }).then(next, next)
})

What if the route doesn't exist?

What if a user GET /thing/:id/lakjsdflkajsdlkfjasdf? What would actually happen is:

  1. Match /thing/:id/*
  2. Get req.user
  3. Look for the next match
  4. Return 404 (because there are no matches)

Your app has executed an extra database call when it didn't have to:

  1. Match all routes
  2. Return 404 because none match

Except this doesn't actually really work with wildcard routing.

What if it's more than just a database call?

If you add a multipart parser that downloads disks to files like this:

app.use(bodyParser.multipart())

If an attacker simply posts a bunch of files to ANY route, your server would quickly be flooded with files and run out of disk. This is one of the main reasons the multipart parser was removed from Connect and Express. Because implementing multipart is controversial, it's better not to provide the parser at all.

Only define routes that are actually served

Your routes should really just look like:

app.route('/things')
.get()
.post()
app.route('/thing/:id([0-9a-f]{24})/comments')
.get()
.post()

The router itself knows whether a path 404s and resolves the routes accordingly. Unfortunately, this is annoying to do as your app will quickly look like this:

app.route('/things')
.post(authenticate(), bodyParser.multipart(), function (req, res, next) {
  // only download files af
})

app.route('/things/:id')
.patch(authenticateUser, getThing, bodyParser.json(), function (req, res, next) {
  // update user
})
.get(getThing, function (req, res, next) {
  res.json(req.thing)
})

The Koa way

It becomes very complex and ugly to route properly Express because of the extensive use of callbacks. It's simply not an option to have every route have 10 nested callbacks. But with Koa or any future async/await framework, things will be much easier when your code looks like this:

const parse = require('co-body')

// NOTE: koa doesn't actually provide a router
app.post('/things', function* (next) {
  let user = yield authenticate(this)
  this.assert(user, 401, 'You must be signed in!')
  this.assert(user.permission.post, 403, 'You cannot post!')
  let body = yield parse(this)
  this.assert(body, 400, 'Invalid body!')
  this.assert(body.name, 422, 'Invalid name!')
  let thing = yield Thing.create(body)
  this.body = thing
})

Conclusion

Try to avoid using wildcard routing. If you're using Express, prepare for some possible breaking changes in 5.x. If you're using a trie router, don't even bother with it.

Generator and Promise Tricks July 18, 2014

When ditching callbacks for promises or generators, life suddenly becomes much easier. You don't have to worry as much about error handling. There are no pyramids of doom. But at first, you'll be confused with how promises and generators work, and finally, you'll be able to use them with expertise!

.map(function* () {})

Some people, including me until recently, see function* () {} as a magical function. They don't correlate function* () {} with regular functions. However, function* () {} is in fact a regular function! The only real difference between function* () {} and function () {} is that the former returns a generator. That means you can pass function* () {} basically anywhere a regular function can go, but just make sure you realize that a generator is returend.

For example, to execute in parallel, you might do something like this:

co(function* () {
  var values = [] // some values
  yield values.map(function (x) {
    return somethingAsync(x)
  })
})()

function* somethingAsync(x) {
  // do something async
  return y
}

When you can just do:

co(function* () {
  var values = []
  yield values.map(somethingAsync)
})

Similarly, you can return promises!

co(function* () {
  var values = []
  var mappedValues = yield values.map(toPromise)
})()

function toPromise(x) {
  return new Promise(function (resolve, reject) {
    resolve(x)
  })
}

Wrapping Generators

As such, you could wrap generator functions in regular functions.

function* wrappedFn(x) {
  return yield* somethingAsync(x)
}

Can really just be:

function wrappedFn(x) {
  return somethingAsync(x)
}

And in both cases, you'll be able to do yield* wrappedFn().

.then = (resolve, reject) =>

One thing I like about the promise specification is that you could literally convert anything into a promise. A promise is just any object with .then().

For example, suppose you have an object stream, and you want .then() to exhaust the stream and return all the objects of the stream.

var toArray = require('stream-to-array')
stream.then = function (resolve, reject) {
  return toArray(this).then(resolve, reject)
}

Now you can simply do stream.then() or even yield stream!

co(function* () {
  var docs = yield stream
})()

Keep in mind that you need to pass resolve and reject to the real .then() method, which you're just proxying to.

Even better, you can create an iterator out of this!

function Iterator() {
  this.i = 0
}

Iterator.prototype.then = function (resolve, reject) {
  return Promise.resolve(this.i++).then(resolve, reject)
}

co(function* () {
  var iterator = new Iterator()
  var i
  while (100 > i = yield iterator) {
    // do something
  }
})()

Basically, just tack on a .then() method to any object you'd like, and you've created a 'yieldable'!

Why You Should and Shouldn't Use Koa May 5, 2014

There's a new node.js framework in town, and its name is Koa. It's the spiritual successor to Connect and Express, written by the same author, TJ Holowaychuk. It has a very similar middleware system, but is completely incompatible with any other node.js framework.

Koa is bleeding edge and has not yet reached version 1.0, but many people including TJ and myself have already ditched Express for Koa. TJ himself has stepped back from maintaining Connect and Express and has instead delegated maintenance to a team, myself included. Don't worry about using Connect or Express, they will still be maintained!

So why should you and shouldn't you ditch Express for Koa like TJ and I have?

Why you should

Superior, callback-less control flow

Thanks to Koa's underlying generator engine co, there's no more callback hell. Of course, this is assuming you write your libraries using generators, promises, or return thunks.

But co's control flow handling isn't about eliminating callbacks. You can also execute multiple asynchronous tasks in parallel and in series without calling a function.

app.use(function* () {
  yield [fn1, fn2, fn3]
})

Bam! You've just executed three asynchronous functions in parallel. You've eliminated the need for any other control flow library such as async, and you don't have to require() anything.

Superior middleware error handling

Thanks to co, you can simply use try/catch instead of node's if (err) callback(err) type error handling. You can see this in the error handling examples in Koa:

app.use(function* (next) {
  try {
    yield* next
  } catch (err) {
    console.error('an error occured! writing a response')
    this.response.status = 500
    this.response.body = err.message
  }
})

Instead of adding an error handling middleware via app.use(function (err, req, res, next) {}) which barely works correctly, you can finally simply use try/catch. All errors will be caught, unless you throw errors on different ticks like so:

app.use(function* () {
  setImmediate(function () {
    throw new Error('this will not be caught by koa '
      + 'and will crash your process')
  })
})

Don't do that! Write your code in generators, promises, or unnested callbacks and you'll be fine.

Superior stream handling

Suppose you want to stream a file to the response with gzip. In vanilla node.js, it'll look like this:

http.createServer(function (req, res) {
  // set the content headers
  fs.createReadStream('filename.txt')
  .pipe(zlib.createGzip())
  .pipe(res)
})

However, you haven't handled any errors. It should look more like this:

http.createServer(function (req, res) {
  // set the content headers
  fs.createReadStream('filename.txt')
  .on('error', onerror)
  .pipe(zlib.createGzip())
  .on('error', onerror)
  .pipe(res)

  function onerror(err) {
    console.error(err.stack)
  }
})

But if you used this method, you'll still get memory leaks when clients abort the request. This is because close events on the final destination stream are not propagated through the pipe()s back to the original stream. You need to use something like finished, otherwise you'll leak file descriptors. Thus, your code should look more like:

http.createServer(function (req, res) {
  var stream = fs.createReadStream('filename.txt')

  // set the content headers
  stream
  .on('error', onerror)
  .pipe(zlib.createGzip())
  .on('error', onerror)
  .pipe(res)

  finished(res, function () {
    // make sure the stream is always destroyed
    stream.destroy()
  })
})

Since you've handled all your errors, you wouldn't need to use domains. But look at it. It's so much code just to send a file. Express also does not handle the close event, so you'll always need to use finished as well.

app.use(require('compression')())
app.use(function (req, res) {
  // set content headers
  var stream = fs.createReadStream('filename.txt')
  stream.pipe(res)
  finished(res, function () {
    stream.destroy()
  })
})

How would this look like in Koa?

app.use(require('koa-compress')())
app.use(function* () {
  this.type = 'text/plain'
  this.body = fs.createReadStream('filename.txt')
})

Since you simply pass the stream to Koa instead of directly piping, Koa is able to handle all these cases for you. You won't need to use domains as no uncaught exceptions will ever be thrown. Don't worry about any leaks as Koa handles that for you as well. You may treat streams essentially the same as strings and buffers, which is one of the main philosophies behind Koa's abstractions.

In other words, Koa tries to fix all of node's broken shit. For example, this case is not handled:

app.use(function* () {
  this.body = fs.createReadStream('filename.txt')
    .pipe(zlib.createGzip())
})

Don't ever use .pipe() unless you know what you're doing. It's broken. Let Koa handle streams for you.

Concise code

Writing apps and middleware for Koa is generally much more concise than any other framework. There are many reasons for this.

The first and obvious reason is the use of generators to remove callbacks. You're no longer creating functions everywhere, just yielding. There's no more nested code to deal with.

Many of the small HTTP utilities in the expressjs organization are included with Koa, so when writing applications and middleware, you don't need to install many third party dependencies.

The last and I think the most important reason is that Koa abstract's node's req and res objects, avoiding any "hacks" required to make things work.

Better written middleware

Part of what makes Connect and Express great is its middleware ecosystem. But what I greatly disliked about this ecosystem was that middleware are generally terrible. There are many reasons for this aside from the inverse of the points above.

Express is similar to Koa in that many utilities are included. This should make writing middleware for Express almost as easy as Koa, but if you're writing middleware for Express, you might as well make it compatible with node.js and any other app.use(function (req, res, next) {}) framework. Supporting only Express at that point is silly. However, you'll end up with a lot of tiny dependencies, which is annoying. Koa middleware on the other hand is completely incompatible with node.js.

Express uses node's original req and res objects. Properties have to be overwritten for middleware to work properly. For example, if you look at the compression middleware, you'll see that res.write() and res.end() are being overwritten. In fact, a lot of middleware are written like this. And it's ugly.

Thanks to Koa's abstraction of node's req and res objects, this is not a problem. Look at koa-compress source code and tell me which one is more concise and readable. Unlike Express, the compression stream's errors are actually handled as well and pipe() is actually used internally.

Then there's the fact that asynchronous functions' errors are simply logged instead of handled. Developers are not even given a choice. This is not a problem with Koa! You can handle all the errors!

Although we're going to have to recreate the middleware ecosystem for Koa from the ground up, I believe that all Koa middleware are fundamentally better than any other frameworks'.

Why you shouldn't

Generators are confusing

There are two programming concepts you have to learn to get started with Koa. First is generators, obviously. But generators are actually quite complicated. In fact, any control flow mechanism, including promises, is going to be confusing for beginners. Unlike promises, co is not based on a specification, so you have to learn both how generators work as well as co.

You also need to understand how this works. It becomes much more important when Koa uses this to pass data instead of node's req and res objects. You may want to read yield next vs yield* next.

Generators are not supported out of the box

There are currently two ways to use generators in node.js.

The first is to use v0.11, an unstable version of node, with the --harmony-generators flag, but you're obviously using an unstable version of node.js. For many people and companies, this is unacceptable, especially since many C/C++ addons don't work with v0.11 yet. Since you need to explicitly set the --harmony-generators flag, creating and using executables is also more difficult.

The second way to use generators is by using gnode. The problem with this is that it's really slow. It basically transpiles all files with generators when require()ing. I tried this before, and it took about 15 seconds for my app to even start. This is unacceptable during development.

We're going to have to wait until node v0.14 or v1 to be able to use generators without any flags. Until then, you're going to be inconvenienced one way or another.

Documentation is sparse

Koa is pretty new, and TJ and I just don't have the time to write thorough documents. Some things are still subject to change, so we don't want to be too thorough or else we'd confuse people down the road. It's also radically different than other frameworks, so we'd have to explain both the philosophy as well as the technical details, otherwise developers are going to get lost.

There have been a few blog posts, but in my opinion they don't explain Koa well enough. The goal of this blog post is to explain more of the benefits instead of the philosophy or technical. If you want to know more about the philosophical, watch as I write my Koa talk.