Table of Contents for

Learning Node

In the earlier days of Node, asynchronous functionality was facilitated through the use of promises, a concept that arose in the 1970s. A promise is an object that represents the result of an asynchronous action. It’s also known as a future, a delay, or simply deferred. The CommonJS design model embraced the concept of the promise.

In the earlier Node implementation, a promise was an object that emitted exactly two events: success and error. Its use was simple: if an asynchronous operation succeeded, the success event was emitted; otherwise, the error event was emitted. No other events were emitted, and the object would emit one or the other, but not both, and no more than once. Example 5-1 incorporates a previously implemented promise into a function that opens and reads in a file.

function test_and_load(filename) {
  var promise = new process.Promise();
  fs.stat(filename).addCallback(function (stat) {

    // Filter out non-files
    if (!stat.isFile()) { promise.emitSuccess(); return; }

    // Otherwise read the file in
    fs.readFile(filename).addCallback(function (data) {
      promise.emitSuccess(data);
    }).addErrback(function (error) {
      promise.emitError(error);
    });

  }).addErrback(function (error) {
    promise.emitError(error);
  });
  return promise;
}

Each object would return the promise object. The code to process a successful result would be passed as a function to the promise object’s addCallback method, which had one parameter, the data. The code to process the error would be passed as a function to the promise object’s addErrback method, which received the error as its one and only parameter:

var File = require('file');
var promise = File.read('mydata.txt');
promise.addCallback(function (data) {
  // process data
});
promise.addErrback(function (err) {
  // deal with error
})

The promise object ensured that the proper functionality was performed whenever the event finished—either the results could be manipulated, or the error processed.

The promise object was pulled from Node in version 0.1.30. As Ryan Dahl noted at the time, the reasoning was:

Rather than the promise object, Node incorporated the last argument callbacks we’ve used in previous chapters. All asynchronous methods feature a callback function as the last argument. The first argument in this callback function is always an error object.

To demonstrate the fundamental structure of the callback functionality, Example 5-2 is a complete Node application that creates an object with one method, someMethod. This method takes three arguments, the second of which must be a string, and the third being the callback. In the method, if the second argument is missing or is not a string, the object creates a new Error object, which is passed to the callback function. Otherwise, whatever the result of the method is gets passed to the callback function.

var obj =function() { };

obj.prototype.doSomething = function(arg1, arg2_) {
  var arg2 = typeof(arg2_) === 'string' ? arg2_ : null;

  var callback_ = arguments[arguments.length - 1];
  callback = (typeof(callback_) == 'function' ? callback_ : null);

  if (!arg2)
     return callback(new Error('second argument missing or not a string'));

  callback(arg1);
}
var test = new obj();

try {
   test.doSomething('test', 3.55, function(err,value) {
      if (err) throw err;

      console.log(value);
   });
} catch(err) {
   console.error(err);
}

The key elements of the callback functionality are in boldface in the code.

The first key functionality is to ensure the last argument is a callback function. Well, we can’t determine the user’s intent, but we can make sure the last argument is a function, and that will have to do. The second key functionality is to create the new Node Error object if an error occurs, and return it as the result to the callback function. The last critical functionality is to invoke the callback function, passing in the method’s result if no error occurs. In short, everything else is changeable, as long as these three key functionalities are present:

With the existing code in Example 5-1, the application output is the following error message printed out to the console:

[Error: second argument missing or not a string]

Changing the method call in the code to the following:

   test.doSomething('test','this',function(err,value) {

results in test being printed out to the console. Changing it then to the following:

   test.doSomething('test',function(err,value) {

again results in an error, this time because the second argument is missing.

If you look through the code in the lib directory of the Node installation, you’ll see the last callback pattern repeated throughout. Though the functionality may change, this pattern remains the same.

This approach is quite simple and ensures consistent results from asynchronous methods. However, it also creates its own unique challenges, as we’ll cover in the next section.

It’s not unusual to find the following in a client-side JavaScript application:

val1 = callFunctionA();
val2 = callFunctionB(val1);
val3 = callFunctionC(val2);

The functions are called, in turn, passing the results from the earlier function to each subsequent function. Since all the functions are synchronous, we don’t have to worry about the function calls getting out of sequence—no unexpected results.

Example 5-3 shows a relatively common case of this type of sequential programming. The application uses synchronous versions of Node’s File System methods to open a file and get its data, modify the data by replacing all references to “apple” with “orange,” and output the resulting string to a new file.

var fs = require('fs');

try {
   var data = fs.readFileSync('./apples.txt','utf8');
   console.log(data);
   var adjData = data.replace(/[A|a]pple/g,'orange');

   fs.writeFileSync('./oranges.txt', adjData);
} catch(err) {
  console.error(err);
}

Since problems can occur and we can’t be sure errors are handled internally in any module function, we wrap all of the function calls in a try block to allow for graceful—or at least, more informative—exception handling. The following is an example of what the error looks like when the application can’t find the file to read:

{ [Error: ENOENT, no such file or directory './apples.txt']
  errno: 34,
  code: 'ENOENT',
  path: './apples.txt',
  syscall: 'open' }

While perhaps not very user-friendly, at least it’s a lot better than the alternative:

node.js:201
        throw e; // process.nextTick error, or 'error' event on first tick
              ^
Error: ENOENT, no such file or directory './apples.txt'
    at Object.openSync (fs.js:230:18)
    at Object.readFileSync (fs.js:120:15)
    at Object.<anonymous> (/home/examples/public_html/node/read.js:3:18)
    at Module._compile (module.js:441:26)
    at Object..js (module.js:459:10)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)
    at Array.0 (module.js:479:10)
    at EventEmitter._tickCallback (node.js:192:40)

In the example, we’re going to have expected results because each function call is performed in sequence.

Converting this synchronous sequential application pattern to an asynchronous implementation requires a couple of modifications. First, we have to replace all functions with their asynchronous counterparts. However, we also have to account for the fact that each function doesn’t block when called, which means we can’t guarantee the proper sequence if the functions are called independently of each other. The only way to ensure that each function is called in its proper sequence is to use nested callbacks.

Example 5-4 is an asynchronous version of the application from Example 5-3. All of the File System function calls have been replaced by their asynchronous versions, and the functions are called in the proper sequence via a nested callback.

var fs = require('fs');

try {
   fs.readFile('./apples2.txt','utf8', function(err,data) {

     if (err) throw err;

     var adjData = data.replace(/[A|a]pple/g,'orange');

     fs.writeFile('./oranges.txt', adjData, function(err) {

        if (err) throw err
     });
  });
} catch(err) {
  console.error(err);
}

In Example 5-4, the input file is opened and read, and only when both actions are finished does the callback function passed as the last parameter get called. In this function, the error is checked to make sure it’s null. If not, the error is thrown for catching in the outer exception-handling block.

If no error occurs, the data is processed and the asynchronous writeFile method is called. Its callback function has only one parameter, the error object. If it’s not null, it’s thrown for handling in the outer exception block.

If an error occurred, it would look similar to the following:

/home/examples/public_html/node/read2.js:11
        if (err) throw err;
                       ^
Error: ENOENT, no such file or directory './boogabooga/oranges.txt'

If you want the stack trace of the error, you can print out the stack property of the Node error object:

catch(err) {
   console.log(err.stack);
}

Including another sequential function call adds another level of callback nesting. In Example 5-5, we access a listing of files for a directory. In each of the files, we replace a generic domain name with a specific domain name using the string replace method, and the result is written back to the original file. A log is maintained of each changed file, using an open write stream.

var fs = require('fs');

var writeStream = fs.createWriteStream('./log.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});

try {
   // get list of files
   fs.readdir('./data/', function(err, files) {

      // for each file
      files.forEach(function(name) {

         // modify contents
         fs.readFile('./data/' + name,'utf8', function(err,data) {

            if (err) throw err;
            var adjData = data.replace(/somecompany\.com/g,'burningbird.net');

            // write to file
            fs.writeFile('./data/' + name, adjData, function(err) {

               if (err) throw err;

               // log write
               writeStream.write('changed ' + name + '\n', 'utf8', function(err) {

                  if(err) throw err;
               });
            });
         });
      });
   });
} catch(err) {
  console.error(util.inspect(err));
}

Though the application looks like it’s processing each file individually before moving on to the next, remember that each of the methods used in this application is asynchronous. If you run the application several times and check the log.txt file, you’ll see that the files are processed in a different, seemingly random order. In my data subdirectory I had five files. Running the application three times in a row resulted in the following output to log.txt (blank lines inserted for clarity):

changed data1.txt
changed data3.txt
changed data5.txt
changed data2.txt
changed data4.txt

changed data3.txt
changed data1.txt
changed data5.txt
changed data2.txt
changed data4.txt

changed data1.txt
changed data3.txt
changed data5.txt
changed data4.txt
changed data2.txt

Another issue arises if you want to check when all of the files have been modified in order to do something. The forEach method invokes the iterator callback functions asynchronously, so it doesn’t block. Adding a statement following the use of forEach, like the following:

console.log('all done');

doesn’t really mean the application is all finished, just that the forEach method didn’t block. If you add a console.log statement at the same time you log the changed file:

               writeStream.write('changed ' + name + '\n', 'utf8', function(err) {

                  if(err) throw err;
                  console.log('finished ' + name);
               });

and add the following after the forEach method call:

console.log('all finished');

you’ll actually get the following console output:

all done
finished data3.txt
finished data1.txt
finished data5.txt
finished data2.txt
finished data4.txt

To solve this challenge, add a counter that is incremented with each log message and then checked against the file array’s length to print out the “all done” message:

// before accessing directory
var counter = 0;
...
               writeStream.write('changed ' + name + '\n', 'utf8', function(err) {

                  if(err) throw err;
                  console.log('finished ' + name);
                  counter++;
                  if (counter >= files.length)
                    console.log('all done');

               });

You’d then get the expected result: an “all done” message displays after all the files have been updated.

The application works quite well—except if the directory we’re accessing has subdirectories as well as files. If the application encounters a subdirectory, it spits out the following error:

/home/examples/public_html/node/example5.js:20
            if (err) throw err;
                           ^
Error: EISDIR, illegal operation on a directory

Example 5-6 prevents this type of error by using the fs.stats method to return an object representing the data from a Unix stat command. This object contains information about the object, including whether it’s a file or not. The fs.stats method is, of course, another asynchronous method, requiring yet more callback nesting.

var fs = require('fs');

var writeStream = fs.createWriteStream('./log.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});

try {
   // get list of files
   fs.readdir('./data/', function(err, files) {

      // for each file
      files.forEach(function(name) {

         // check to see if object is file
         fs.stat('./data/' + name, function(err, stats) {

            if (err) throw err;

            if (stats.isFile())

               // modify contents
               fs.readFile('./data/' + name,'utf8', function(err,data) {

                  if (err) throw err;
                  var adjData = data.replace(/somecompany\.com/g,'burningbird.net');

                  // write to file
                  fs.writeFile('./data/' + name, adjData, function(err) {

                     if (err) throw err;

                     // log write
                     writeStream.write('changed ' + name + '\n', 'utf8',
                                                                 function(err) {
                        if(err) throw err;
                     });
                  });
               });
         });
      });
   });
} catch(err) {
  console.error(err);
}

Again, the application performs its purpose, and performs it well—but how difficult it is to read and maintain! I’ve heard this type of nested callback called callback spaghetti and the even more colorful pyramid of doom, both of which are apt terms.

The nested callbacks continue to push against the right side of the document, making it more difficult to ensure we have the right code in the right callback. However, we can’t break the callback nesting apart because it’s essential that the methods be called in turn:

What we’d like to do is find a way of implementing this series of method calls but without having to depend on nested callbacks. For this, we need to look at third-party modules that provide asynchronous control flow.

The application in Example 5-6 is an example of an asynchronous pattern, where each function is called in turn and passes its results to the next function, and the entire chain stops only if an error occurs. There are several such patterns, though some are variations of others, and not everyone uses the exact same terminology.

One Node module, Async, provides names and support for the most extensive list of asynchronous control flow patterns:

In the listing of modules provided at the Node.js website, there is a category titled “Control Flow/Async Goodies.” In this list is the Async module, which provides the asynchronous control patterns I just listed. Though not every control flow module provides the capability to handle all possible patterns, most provide functionality for the most common patterns: series (also called sequence and sometimes referred to as waterfall—as in the preceding list—though Async lists waterfall separately from series) and parallel. In addition, some of the modules also reinstate the concept of promises from earlier editions of Node, while others implement a concept called fibers, which emulate threads.

In the next couple of sections, I’ll demonstrate two of the more popular of the actively maintained control flow modules: Step and Async. Each offers its own unique perspective on asynchronous control flow management, though both provide a very useful—and likely essential—service.

npm install step

var fs = require('fs'),
    Step = require('step');

try {

   Step (
      function readData() {
          fs.readFile('./data/data1.txt', 'utf8', this);
      },
      function modify(err, text) {
         if (err) throw err;
         return text.replace(/somecompany\.com/g,'burningbird.net');
      },
      function writeData(err, text) {
          if (err) throw err;
          fs.writeFile('./data/data1.txt', text, this);
      }
    );
} catch(err) {
   console.error(err);
}

      function writeFile(err, text) {
          if (err) throw err;
          fs.writeFile('./boogabooga/data/data1.txt');
      }

var fs = require('fs'),
    Step = require('step'),
    files,
    _dir = './data/';

try {

   Step (
      function readDir() {
          fs.readdir(_dir, this);
      },
      function readFile(err, results) {
          if (err) throw err;
          files = results;
          var group = this.group();
          results.forEach(function(name) {
             fs.readFile(_dir + name, 'utf8', group());
          });
      },
      function writeAll(err, data) {
         if (err) throw err;
         for (var i = 0; i < files.length; i++) {
           var adjdata = data[i].replace(/somecompany\.com/g,'burningbird.net');
           fs.writeFile(_dir + files[i], adjdata, 'utf8',this);
         }
      }
    );
} catch(err) {
   console.log(err);
}

var fs = require('fs'),
    Step = require('step'),
    files;

try {

   Step (
      function readFiles() {
         fs.readFile('./data/data1.txt', 'utf8',this.parallel());
         fs.readFile('./data/data2.txt', 'utf8',this.parallel());
         fs.readFile('./data/data3.txt', 'utf8',this.parallel());
      },
      function writeFiles(err, data1, data2, data3) {
         if (err) throw err;
         data1 = data1.replace(/somecompany\.com/g,'burningbird.net');
         data2 = data2.replace(/somecompany\.com/g,'burningbird.net');
         data3 = data3.replace(/somecompany\.com/g,'burningbird.net');

         fs.writeFile('./data/data1.txt', data1, 'utf8', this.parallel());
         fs.writeFile('./data/data2.txt', data2, 'utf8', this.parallel());
         fs.writeFile('./data/data3.txt', data3, 'utf8', this.parallel());
      }
    );
} catch(err) {
   console.log(err);
}

npm install async

var fs = require('fs'),
    async = require('async');

try {
   async.waterfall([
      function readData(callback) {
         fs.readFile('./data/data1.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      function modify(text, callback) {
         var adjdata=text.replace(/somecompany\.com/g,'burningbird.net');
         callback(null, adjdata);
      },
      function writeData(text, callback) {
          fs.writeFile('./data/data1.txt', text, function(err) {
             callback(err,text);
          });
      }
    ], function (err, result) {
         if (err) throw err;
         console.log(result);
    });
} catch(err) {
   console.log(err);
}

var fs = require('fs'),
    async = require('async'),
    _dir = './data/';

var writeStream = fs.createWriteStream('./log.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});
try {
   async.waterfall([
      function readDir(callback) {
         fs.readdir(_dir, function(err, files) {
            callback(err,files);
         });
      },
      function loopFiles(files, callback) {
         files.forEach(function (name) {
            callback (null, name);
         });
      },
      function checkFile(file, callback) {
         fs.stat(_dir + file, function(err, stats) {
            callback(err, stats, file);
         });
      },
      function readData(stats, file, callback) {
         if (stats.isFile())
            fs.readFile(_dir + file, 'utf8', function(err, data){
              callback(err,file,data);
          });
      },
      function modify(file, text, callback) {
         var adjdata=text.replace(/somecompany\.com/g,'burningbird.net');
         callback(null, file, adjdata);
      },
      function writeData(file, text, callback) {
          fs.writeFile(_dir + file, text, function(err) {
             callback(err,file);
          });
      },
      function logChange(file, callback) {
          writeStream.write('changed ' + file + '\n', 'utf8', function(err) {
             callback(err, file);
         });
      }
    ], function (err, result) {
         if (err) throw err;
         console.log('modified ' + result);
    });
} catch(err) {
   console.log(err);
}

var fs = require('fs'),
    async = require('async');

try {
   async.parallel({
      data1 : function (callback) {
         fs.readFile('./data/data1.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      data2 : function (callback) {
         fs.readFile('./data/data2.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      data3 : function readData3(callback) {
         fs.readFile('./data/data3.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },

    }, function (err, result) {
         if (err) throw err;
         console.log(result);
    });
} catch(err) {
   console.log(err);
}

{ data1: 'apples\n', data2: 'oranges\n', data3: 'peaches\n' }

A couple of times in the chapter I mentioned people recommending certain restraints, such as using named rather than anonymous functions in Node applications. Collectively, these restraints are known as preferred Node style, though there is no one style guide or definitive set of shared preferences. In fact, there are several different recommendations for proper Node style.

Here are some of the recommendations, and my own take on each:

The most important rule to remember out of all of these is to use asynchronous functions whenever and wherever possible. After all, asynchronous functionality is the heart of Node.