Alternatives to Deeply-Nested Callback Functions in Javascript

A few days ago, I wrote a server in Node.js, the purpose of which is to email me when an API found in Nuget is updated. I thought the project would take a few hours, but as it turned out, it took several days. You may ask “How is this possible for such an easy problem?” Unfortunately, I spent a lot of time trying to figure out what API to use to read a text file, and how to use the API. It led me to an understanding of a fundamental problem with Javascript known as “callback hell” (1, 2).

In languages like C# and Java, file system I/O requests are usually blocking: the function does not return until the request is complete. A call to System.IO.File.ReadAllText opens a file, reads all the lines of text in the file into a string, closes the file, and returns the string. Control flow is transferred to the function and does not return until completion. Blocking functions are easy to use and understand in an imperative or functional programming context.

However, in Javascript, most APIs are non-blocking in order to boost performance and responsiveness of an application. Non-blocking function calls are implemented using a callback function, a computation that is executed by another function. For example, fs.readFile is a non-blocking function that opens a file, reads the text in the file, calls a callback function with two arguments, one of which contains the text, and closes the file. Code that textually follows fs.readFile cannot depend on the data because the callback is invoked asynchronously. Since the data is available through the callback function, a sequence of callback functions invocations is required, one dependent on the result of an enclosing callback. When callbacks are written as anonymous functions, the dependency is a nesting.

Deeply-nested callback functions

In my project, I wanted to create an array of URLs for APIs in Nuget.org from a file. Each URL in the file is on a separate line, and may contain trailing spaces, tabs, new lines, and carriage returns. Using nested callback functions, the Javascript code for this may look like the following:

var fs = require('fs');
var file_path = "files.txt";
fs.access(file_path, fs.R_OK, (err) => {
    console.log(err ? 'no access!' : 'can read');
    if (err != null)
    {
        console.log("File " + file_path + " does not exist.");
        process.exit();
    }
    else
    {
        fs.readFile(file_path, 'utf-8', (err, data) => {
            if (err) throw err;
            var where = data
                .replace(/[\r\t]+/g, '')
                .split('\n');
            var linq = require('linq-iterator');
            where = linq.from(where).select(y => y.trim()).toArray();
            console.log("All addr" + where);
        });
    }
});

As we implement the code that further processes the list of URLs (to request the webpage and extract dates of when the API was updated), the depth of the nesting callback functions increases. Callback chains is an implementation of a DU-chain of asynchronous variables, which is, unfortunately, the result of the design of Javascript.  At some point the code becomes unmanageable (3).

Alternatives to deeply-nested callback functions

Over the years, a number of solutions have been developed to solve deep nesting (4-9).

Declare a function for each nesting

Perhaps the easiest solution is to place each block corresponding to the callback in a separate function. The advantage of this that each block is now at the same lexical level.

var fs = require('fs');
var file_path = "files.txt";

fs.access(file_path, fs.R_OK, func1);

function func1(err)
{
    console.log(err ? 'no access!' : 'can read');
    if (err != null)
    {
        console.log("File " + file_path + " does not exist.");
        process.exit();
    }
    else
    {
        fs.readFile(file_path, 'utf-8', func2);
    }
}

function func2(err, data)
{
    if (err) throw err;
    var where = data
	.replace(/[\r\t]+/g, '')
	.split('\n');
    var linq = require('linq-iterator');
    where = linq.from(where).select(y => y.trim()).toArray();
    console.log("All addr" + where);
}

One must judiciously decide between placing short blocks into separate functions that may be far removed from the context of the enclosing/calling block. However, this solution does not really solve the main problem of long chains of callback functions: a function is still a function regardless if it is has a name or is anonymous.

Function chaining using Promises

Promises for Javascript is an extension of a callback (10, 11). Functions that return a Promise can be chained together using the “then” method at the same lexical level.

var fs = require('fs');
var file_path = "files.txt";

new Promise((resolve, reject) => {
    fs.access(file_path, fs.R_OK, (err) => {
        console.log(err ? 'no access!' : 'can read');
        if (err != null) {
            console.log("File " + file_path + " does not exist.");
            reject("fucked.");
        } else {
            resolve();
        }
    })
}).then(() => {
    return new Promise((resolve, reject) => {
        fs.readFile(file_path, 'utf-8', (err, data) => {
            if (err) reject();
            var where = data
                .replace(/[\r\t]+/g, '')
                .split('\n');
            var linq = require('linq-iterator');
            where = linq.from(where).select(y => y.trim()).toArray();
            return resolve(where);
        });
    });
}).then((value) => {
    list = value;
    console.log("list = " + list);
    return value;
});

Support for Asynchronous Programming

Support for asynchronous operations within the language are available for many languages, including Javascript in ES8/ES2017. (NB: async/await was dropped from ES7/ES2016. See the spec.) The async and await keywords eliminate the need for explicit callback functions. With the await function, continuation is registered as a callback with the awaiting expression. async/await is a feature of ES8, but is available in Node.js using Babel. (12-16)

Await/Async in ES8/ES2017

This solution works with Node.js 4.6+ and Babel. To set up Webstorm, follow these instructions.

require("babel-core/register");
require("babel-polyfill");

var fs = require('fs');
var file_path = "files.txt";

const foo = function() {
    return new Promise((resolve, reject) => {
        fs.access(file_path, fs.R_OK, (err) => {
            console.log(err ? 'no access!' : 'can read');
            if (err != null) {
                console.log("File " + file_path + " does not exist.");
                reject("fucked.");
            } else {
                resolve();
            }
        })
    });
};

var bar = function() {
    return new Promise((resolve, reject) => {
        fs.readFile(file_path, 'utf-8', (err, data) => {
            if (err) reject();
            var where = data
                .replace(/[\r\t]+/g, '')
                .split('\n');
            var linq = require('linq-iterator');
            where = linq.from(where).select(y => y.trim()).toArray();
            return resolve(where);
        });
    });
};

var driver = async function()
{
    var x = await foo();
    var list = await bar();
    console.log("list = " + list);
};

driver();
Asyncawait API

A similar solution is available for ES6 Javascript using the API “asyncawait”, available through NPM. It provides a similar syntax and works well.

var async = require('asyncawait/async');
var await = require('asyncawait/await');
var Promise = require('bluebird');
var fs = Promise.promisifyAll(require('fs'));

var file_path = "files.txt";

var exists = async(function() {
    await(fs.accessAsync(file_path, fs.R_OK));
    });

exists()
    .then(() => {
        console.log("ok");
        })
    .catch((err) => {
        console.log("doesn't exist.");
        process.exit();
        });

var where = async(function () {
    var data = await(fs.readFileAsync(file_path, 'utf-8'));
    var where = data
        .replace(/[\r\t]+/g, '')
        .split('\n');
    var linq = require('linq-iterator');
    where = linq.from(where).select(y => y.trim()).toArray();
    return where;
    });

where().then((list) => {
    console.log("list = " + list);
    });

Further Information

  1. Callback Hell. http://callbackhell.com/
  2. Pyramid of doom (programming). https://en.wikipedia.org/wiki/Pyramid_of_doom_(programming).
  3. Harrison, Warren, and Curtis Cook. “Are deeply nested conditionals less readable?.” Journal of Systems and Software 6.4 (1986): 335-341. http://www.sciencedirect.com/science/article/pii/0164121286900038
  4. Computer Programming/Coding Style/Minimize nesting. https://en.wikibooks.org/wiki/Computer_Programming/Coding_Style/Minimize_nesting
  5. Handling Synchronous Asynchronous Loops In Javascript/Node.JS. https://zackehh.com/handling-synchronous-asynchronous-loops-javascriptnode-js/
  6. Green, Thomas R. G. “Ifs and thens: Is nesting just for the birds?.” Software: Practice and Experience 10.5 (1980): 373-381. http://onlinelibrary.wiley.com/doi/10.1002/spe.4380100505/abstract
  7. Clarke, Lori A., Jack C. Wileden, and Alexander L. Wolf. “Nesting in Ada programs is for the birds.” ACM Sigplan Notices. Vol. 15. No. 11. ACM, 1980. http://dl.acm.org/citation.cfm?id=948651
  8. https://www.quora.com/How-do-you-make-an-async-function-synchronous
  9. http://stackabuse.com/avoiding-callback-hell-in-node-js/
  10. Futures and promises. https://en.wikipedia.org/wiki/Futures_and_promises
  11. https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise
  12. http://stackabuse.com/node-js-async-await-in-es7/
  13. http://es6-features.org/
  14. Bierman, Gavin, et al. “Pause’n’Play: Formalizing Asynchronous C^\ sharp.”European Conference on Object-Oriented Programming. Springer Berlin Heidelberg, 2012. http://link.springer.com/chapter/10.1007/978-3-642-31057-7_12
  15. Tasirlar, Sagnak, and Vivek Sarkar. “Data-driven tasks and their implementation.” 2011 International Conference on Parallel Processing. IEEE, 2011. http://dl.acm.org/citation.cfm?id=2066922
  16. Cordemans, Piet, Eric Steegmans, and Jeroen Boydens. “Task Parallel Paradigms: a Comparative Case Study.” Annual Journal of Electronics. Vol. 7. Technical Univ. of Sofia, 2013. https://lirias.kuleuven.be/handle/123456789/416602
  17. Callback heaven for Node.js with async/await. https://github.com/yortus/asyncawait