Before installing Node in Linux, you need to prepare your environment. As noted in the documentation provided in the Node wiki, first make sure Python is installed, and then install libssl-dev if you plan to use SSL/TLS (Secure Sockets Layer/Transport Layer Security). Depending on your Linux installation, Python may already be installed. If not, you can use your systems package installer to install the most stable version of Python available for your system, as long as it’s version 2.6 or 2.7 (required for the most recent version of Node).

For both Ubuntu and Debian, you’ll also need to install other libraries. Using the Advanced Packaging Tool (APT) available in most Debian GNU/Linux systems, you can ensure the libraries you need are installed with the following commands:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential openssl libssl-dev pkg-config

The update command just ensures the package index on your system is up to date, and the upgrade command upgrades any existing outdated packages. The third command line is the one that installs all of the necessary packages. Any existing package dependencies are pulled in by the package manager.

Once your system is prepared, download the Node tarball (the compressed, archived file of the source) to your system. I use wget to access tarballs, though you can also use curl. At the time I’m writing this, the most recent source for Node is version 0.8.2:

wget http://nodejs.org/dist/v0.8.2/node-v0.8.2.tar.gz

Once you’ve downloaded it, unzip and untar the file:

tar -zxf node-v0.8.2.tar.gz

You now have a directory labeled node-v0.6.18. Change into the directory and issue the following commands to compile and install Node:

./configure
make
sudo make install

If you’ve not used the make utility in Unix before, these three commands set up the makefile based on your system environment and installation, run a preliminary make to check for dependencies, and then perform a final make with installation. After processing these commands, Node should now be installed and accessible globally via the command line.

Notice in the last command that you had to use sudo to install Node. You need root privileges to install Node this way (see the upcoming note). However, you can install Node locally by using the following, which installs Node in a given local subdirectory:

mkdir ~/working
./configure --prefix=~/working
make
make install
echo 'export PATH=~/working/bin:${PATH}' >> ~/.bashrc
. ~/.bashrc

So, as you can see here, setting the prefix configuration option to a specified path in your home directory installs Node locally. You’ll need to remember to update your PATH environmental variable accordingly.

Although you can install Node locally, if you’re thinking of using this approach to use Node in your shared hosting environment, think again. Installing Node is just one part of using Node in an environment. You also need privileges to compile an application, as well as run applications off of certain ports (such as port 80). Most shared hosting environments will not allow you to install your own version of Node.

Unless there’s a compelling reason, I recommend installing Node using sudo.

You can install Node in Windows using a very basic installation sequence as outlined in the wiki installation page provided earlier. However, chances are that if you’re going to use Node in a Windows environment, you’re going to use it as part of a Windows web development infrastructure.

There are two different Windows infrastructures you can use Node with at this time. One is the new Windows Azure cloud platform, which allows developers to host applications in a remote service (called a cloud). Microsoft provides instructions for installing the Windows Azure SDK for Node, so I won’t be covering that process in this chapter (though I will examine and demonstrate the SDK later in the book).

The other approach to using Node on Windows—in this case, Windows 7—is by integrating Node into Microsoft’s WebMatrix, a tool for web developers integrating open source technologies. Here are the steps we’ll need to take to get Node up and running with WebMatrix in Windows 7:

You install WebMatrix using the Microsoft Web Platform Installer, as shown in Figure 1-1. The tool also installs IIS Express, which is a developer version of Microsoft’s web server. Download WebMatrix from http://www.microsoft.com/web/webmatrix/.

Figure 1-1. Installing WebMatrix in Windows 7

Once the WebMatrix installation is finished, install the latest version of Node using the installer provided at the primary Node site (http://nodejs.org/#download). Installation is one-click, and once you’re finished you can open a Command window and type node to check for yourself that the application is operational, as shown in Figure 1-2.

For Node to work with IIS in Windows, install iisnode, a native IIS 7.x module created and maintained by Tomasz Janczuk. As with Node, installation is a snap using the prebuilt installation package, available at https://github.com/tjanczuk/iisnode. There are x86 and x64 installations, but for x64, you’ll need to install both.

Figure 1-2. Testing in the Command window to ensure Node is properly installed

During the iisnode installation, a window may pop up telling you that you’re missing the Microsoft Visual C++ 2010 Redistributable Package, as shown in Figure 1-3. If so, you’ll need to install this package, making sure you get the one that matches the version of iisnode you’re installing—either the x86 package (available at http://www.microsoft.com/download/en/details.aspx?id=5555) or the x64 package (available at http://www.microsoft.com/download/en/details.aspx?id=14632), or both. Once you’ve installed the requisite package, run the iisnode installation again.

Figure 1-3. Message warning us that we need to install the C++ redistributable package

If you want to install the iisnode samples, open a Command window with administrator privileges, go to the directory where iisnode is installed—either Program Files for 64-bit, or Program Files (x86)—and run the setupsamples.bat file.

To complete the WebMatrix/Node setup, download and install the Node templates for WebMatrix, created by Steve Sanderson and found at https://github.com/SteveSanderson/Node-Site-Templates-for-WebMatrix.

You can test that everything works by running WebMatrix, and in the opening pages, select the “Site from Template” option. In the page that opens, shown in Figure 1-4, you’ll see two Node template options: one for Express (introduced in Chapter 7) and one for creating a basic, empty site configured for Node. Choose the latter option, giving the site a name of First Node Site, or whatever you want to use.

Figure 1-4. Creating a new Node site using a template in WebMatrix

Figure 1-5 shows WebMatrix once the site has been generated. Click the Run button, located in the top left of the page, and a browser page should open with the ubiquitous “Hello, world!” message displayed.

If you’re running the Windows Firewall, the first time you run a Node application, you may get a warning like that shown in Figure 1-6. You need to let the Firewall know this application is acceptable by checking the “Private networks” option and then the “Allow access” button. You want to restrict communication to just your private network on your development machine.

Figure 1-5. Newly generated Node site in WebMatrix

Figure 1-6. Warning that the Windows Firewall blocked Node application, and the option to bypass

If you look at the generated files for your new WebMatrix Node project, you’ll see one named app.js. This is the Node file, and it contains the following code:

var http = require('http');

http.createServer(function (req, res) {

    res.writeHead(200, { 'Content-Type': 'text/html' });
    res.end('Hello, world!');

}).listen(process.env.PORT || 8080);

What this all means, I’ll get into in the second part of this chapter. The important item to take away from this code right now is that we can run this same application in any operating system where Node is installed and get the exact same functionality: a service that returns a simple message to the user.

Node stable releases are even numbered, such as the current 0.8.x, while the development releases are odd numbered (currently 0.9.x). I recommend sticking with stable releases only—at least until you have some experience with Node.

Updating your Node installation isn’t complicated. If you used the package installer, using it for the new version should just override the old installation. If you’re working directly with the source, you can always uninstall the old source and install the new if you’re concerned about potential clutter or file corruption. In the Node source directory, just issue the uninstall make option:

make uninstall

Download the new source, compile it, and install it, and you’re ready to go again.

The challenge with updating Node is determining whether a specific environment, module, or other application works with the new version. In most cases, you shouldn’t have version troubles. However, if you do, there is an application you can use to “switch” Node versions. The application is the Node Version Manager (Nvm).

You can download Nvm from GitHub, at https://github.com/creationix/nvm. Like Node, Nvm must be compiled and installed on your system.

To install a specific version of Node, install it with Nvm:

nvm install v0.4.1

To switch to a specific version, use the following:

nvm run v0.4.1

To see which versions are available, use:

nvm ls

As is typical for testing out any new development environment, language, or tool, the first application we’ll create is “Hello, World”—a simple application that prints out a greeting to whomever accesses it.

Example 1-1 shows all the text needed to create Hello, World in Node.

// load http module
var http = require('http');

// create http server
http.createServer(function (req, res) {

  // content header
  res.writeHead(200, {'content-type': 'text/plain'});

  // write message and signal communication is complete
  res.end("Hello, World!\n");
}).listen(8124);

console.log('Server running on 8124');

The code is saved in a file named helloworld.js. As server-side functionality goes, this Node application is neither too verbose, nor too cryptic; one can intuit what’s happening, even without knowing Node. Best of all, it’s familiar since it’s written in a language we know well: JavaScript.

To run the application, from the command line in Linux, the Terminal window in Mac OS, or the Command window in Windows, type:

node helloworld.js

The following is printed to the command line once the program has successfully started:

Server running at 8124

Now, access the site using any browser. If the application is running on your local machine, you’ll use localhost:8124. If it’s running remotely, use the URL of the remote site, with the 8124 port. A web page with the words “Hello, World!” is displayed. You’ve now created your first complete and working Node application.

Since we didn’t use an ampersand (&) following the node command—telling the application to run in the background—the application starts and doesn’t return you to the command line. You can continue accessing the application, and the same words get displayed. The application continues until you type Ctrl-C to cancel it, or otherwise kill the process.

If you want to run the application in the background within a Linux system, use the following:

node helloworld.js &

However, you’ll then have to find the process identifier using ps -ef, and manually kill the right process—in this case, the one with the process identifier 3747—using kill:

ps -ef | grep node
kill 3747

Exiting the terminal window will also kill the process.

You won’t be able to start another Node application listening at the same port: you can run only one Node application against one port at a time. If you’re running Apache at port 80, you won’t be able to run the Node application at this port, either. You must use a different port for each application.

You can also add helloworld.js as a new file to the existing WebMatrix website you created earlier, if you’re using WebMatrix. Just open the site, choose the “New File...” option from the menu bar, and add the text shown in Example 1-1 to the file. Then click the Run button.

I’ll get more in depth on the anatomy of Node applications in the next couple of chapters, but for now, let’s take a closer look at the Hello, World application.

Returning to the text in Example 1-1, the first line of code is:

var http = require('http');

Most Node functionality is provided through external applications and libraries called modules. This line of JavaScript loads the HTTP module, assigning it to a local variable. The HTTP module provides basic HTTP functionality, enabling network access of the application.

The next line of code is:

http.createServer(function (req, res) { ...

In this line of code, a new server is created with createServer, and an anonymous function is passed as the parameter to the function call. This anonymous function is the requestListener function, and has two parameters: a server request (http.ServerRequest) and a server response (http.ServerResponse).

Within the anonymous function, we have the following line:

res.writeHead(200, {'content-Type': 'text/plain'});

The http.ServerResponse object has a method, writeHead, that sends a response header with the response status code (200), as well as provides the content-type of the response. You can also include other response header information within the headers object, such as content-length or connection:

{ 'content-length': '123',
  'content-type': 'text/plain',
  'connection': 'keep-alive',
  'accept': '*/*' }

A second, optional parameter to writeHead is a reasonPhrase, which is a textual description of the status code.

Following the code to create the header is the command to write the “Hello, World!” message:

res.end("Hello, World!\n");

The http.ServerResponse.end method signals that the communication is finished; all headers and the response body have been sent. This method must be used with every http.ServerResponse object.

The end method has two parameters:

Both parameters are optional, and the second parameter is required only if the encoding of the string is anything other than utf8, which is the default.

Instead of passing the text in the end function, I could have used another method, write:

res.write("Hello, World!\n");

and then:

res.end();

The anonymous function and the createServer function are both finished on the next line in the code:

}).listen(8124);

The http.Server.listen method chained at the end of the createServer method listens for incoming connections on a given port—in this case, port 8124. Optional parameters are a hostname and a callback function. If a hostname isn’t provided, the server accepts connections to web addresses, such as http://oreilly.com or http://examples.burningbird.net.

The listen method is asynchronous, which means the application doesn’t block program execution, waiting for the connection to be established. Whatever code following the listen call is processed, and the listen callback function is invoked when the listening event is fired—when the port connection is established.

The last line of code is:

console.log('Server running on 8124/');

The console object is one of the objects from the browser world that is incorporated into Node. It’s a familiar construct for most JavaScript developers, and provides a way to output text to the command line (or development environment), rather than to the client.

To demonstrate Node’s asynchronous nature, Example 1-2 modifies the Hello, World application from earlier in the chapter. Instead of just typing out “Hello, World!” it actually opens up the previously created helloworld.js and outputs the contents to the client.

// load http module
var http = require('http');
var fs = require('fs');

// create http server
http.createServer(function (req, res) {

   // open and read in helloworld.js
   fs.readFile('helloworld.js', 'utf8', function(err, data) {

      res.writeHead(200, {'Content-Type': 'text/plain'});
      if (err)
         res.write('Could not find or open file for reading\n');
      else

         // if no error, write JS file to client
         res.write(data);
      res.end();
     });
}).listen(8124, function() { console.log('bound to port 8124');});

console.log('Server running on 8124/');

A new module, File System (fs), is used in this example. The File System module wraps standard POSIX file functionality, including opening up and accessing the contents from a file. The method used is readFile. In Example 1-2, it’s passed the name of the file to open, the encoding, and an anonymous function.

The two instances of asynchronous behavior I want to point out in Example 1-2 are the callback function that’s attached to the readFile method, and the callback function attached to the listen method.

As discussed earlier, the listen method tells the HTTP server object to begin listening for connections on the given port. Node doesn’t block, waiting for the connection to be established, so if we need to do something once the connection is established, we provide a callback function, as shown in Example 1-2.

When the connection is established, a listening event is emitted, which then invokes the callback function, outputting a message to the console.

The second, more important callback instance is the one attached to readFile. Accessing a file is a time-consuming operation, relatively speaking, and a single-threaded application accessed by multiple clients that blocked on file access would soon bog down and be unusable.

Instead, the file is opened and the contents are read asynchronously. Only when the contents have been read into the data buffer—or an error occurs during the process—is the callback function passed to the readFile method called. It’s passed the error (if any), and the data if no error occurs.

In the callback function, the error is checked, and if there is no error, the data is then written out to the response back to the client.

Most people who have developed with JavaScript have done so in client applications, meant to be run by one person at a time in a browser. Using JavaScript in the server may seem odd. Creating a JavaScript application accessed by multiple people at the same time may seem even odder.

Our job is made easier because of the Node event loop and being able to put our trust in asynchronous function calls. However, we’re no longer in Kansas, Dorothy—we are developing for a different environment.

To demonstrate the differences in this new environment, I created two new applications: one as a service, and one to test the new service. Example 1-3 shows the code for the service application.

In the code, a function is called, synchronously, to write out numbers from 1 to 100. Then a file is opened, similar to what happened in Example 1-2, but this time the name of the file is passed in as a query string parameter. In addition, the file is opened only after a timer event.

var http = require('http');
var fs = require('fs');

// write out numbers
function writeNumbers(res) {

  var counter = 0;

  // increment global, write to client
  for (var i = 0; i<100; i++) {
    counter++;
    res.write(counter.toString() + '\n');
   }
}

// create http server
http.createServer(function (req, res) {

   var query = require('url').parse(req.url).query;
   var app = require('querystring').parse(query).file + ".txt";

   // content header
   res.writeHead(200, {'Content-Type': 'text/plain'});

   // write out numbers
   writeNumbers(res);

   // timer to open file and read contents
   setTimeout(function() {

      console.log('opening ' + app);
      // open and read in file contents
      fs.readFile(app, 'utf8', function(err, data) {
         if (err)
            res.write('Could not find or open file for reading\n');
         else {
            res.write(data);
         }
         // response is done
         res.end();
      });
   },2000);
}).listen(8124);

console.log('Server running at 8124');

The loop to print out the numbers is used to delay the application, similar to what could happen if you performed a computationally intensive process and then blocked until the process was finished. The setTimeout function is another asynchronous function, which in turn invokes a second asynchronous function: readFile. The application combines both asynchronous and synchronous processes.

Create a text file named main.txt, containing any text you want. Running the application and accessing the page from Chrome with a query string of file=main generates the following console output:

Server running at 8124/
opening main.txt
opening undefined.txt

The first two lines are expected. The first is the result of running console.log at the end of the application, and the second is a printout of the file being opened. But what’s undefined.txt in the third line?

When processing a web request from a browser, be aware that browsers may send more than one request. For instance, a browser may also send a second request, looking for a favicon.ico. Because of this, when you’re processing the query string, you must check to see if the data you need is being provided, and ignore requests without the data.

So far, all we’ve done is test our Node applications from a browser. This isn’t really putting much stress on the asynchronous nature of the Node application.

Example 1-4 contains the code for a very simple test application. All it does is use the HTTP module to request the example server several times in a loop. The requests aren’t asynchronous. However, we’ll also be accessing the service using the browser as we run the test program. Both, combined, asynchronously test the application.

var http = require('http');

//The url we want, plus the path and options we need
var options = {
   host: 'localhost',
   port: 8124,
   path: '/?file=secondary',
   method: 'GET'
};

var processPublicTimeline = function(response) {
   // finished? ok, write the data to a file
   console.log('finished request');
};

for (var i = 0; i < 2000; i++) {
   // make the request, and then end it, to close the connection
   http.request(options, processPublicTimeline).end();
}

Create the second text file, named secondary.txt. Put whatever you wish in it, but make the contents obviously different from main.txt.

After making sure the Node application is running, start the test application:

node test.js

As the test application is running, access the application using your browser. If you look at the console messages being output by the application, you’ll see it process both your manual and the test application’s automated requests. Yet the results are consistent with what we would expect, a web page with:

Now, let’s mix things up a bit. In Example 1-3, make the counter global rather than local to the loop function, and start the application again. Then run the test program and access the page in the browser.

The results have definitely changed. Rather than the numbers starting at 1 and going to 100, they start at numbers like 2,601 and 26,301. They still print out the next sequential 99 numbers, but the starting value is different.

The reason is, of course, the use of the global counter. Since you’re accessing the same application in the browser manually as the test program is doing automatically, you’re both updating counter. Both the manual and automated application requests are processed, in turn, so there’s no contention for the shared data (a major problem with thread safety in a multithreaded environment), but if you’re expecting a consistent beginning value, you might be surprised.

Now change the application again, but this time remove the var keyword in front of the app variable—“accidentally” making it a global variable. We all have, from time to time, forgotten the var keyword with our client-side JavaScript applications. The only time we get bit by this mistake is if any libraries we’re using are using the same variable name.

Run the test application and access the Node service in your browser a couple of times. Chances are, you’ll end up with the text from the secondary.txt file in your browser window, rather than the requested main.txt file. The reason is that in the time between when you processed the query for the filename and when you actually opened the file, the automatic application modified the app variable. The test application is able to do so because you made an asynchronous function request, basically ceding program control to another request while your request was still mid-process.

REPL has a simple interface with a small set of useful commands. In the preceding section, I mentioned .save. The .save command saves your inputs in the current object context into a file. Unless you specifically created a new object context or used the .clear command, the context should comprise all of the input in the current REPL session:

> .save ./dir/session/save.js

Only your inputs are saved, as if you had typed them directly into a file using a text editor.

Here is the complete list of REPL commands and their purposes:

If you’re working on an application using REPL as an editor, here’s a hint: save your work often using .save. Though current commands are persisted to history, trying to recreate your code from history would be a painful exercise.

Speaking of persistence and history, now let’s go over how to customize both with REPL.

The Node.js website documentation for REPL mentions setting up an environmental variable so you can use REPL with rlwrap. What is rlwrap, and why would you use it with REPL?

The rlwrap utility is a wrapper that adds GNU readline library functionality to command lines that allow increased flexibility with keyboard input. It intercepts keyboard input and provides additional functionality, such as enhanced line editing, as well as a persistent history of commands.

You’ll need to install rlwrap and readline to use this facility with REPL, though most flavors of Unix provide an easy package installation. For instance, in my own Ubuntu system, installing rlwrap was this simple:

apt-get install rlwrap

Mac users should use the appropriate installer for these applications. Windows users have to use a Unix environmental emulator, such as Cygwin.

Here’s a quick and visual demonstration of using REPL with rlwrap to change the REPL prompt to purple:

env NODE_NO_READLINE=1 rlwrap -ppurple node

If I always want my REPL prompt to be purple, I can add an alias to my bashrc file:

alias node="env NODE_NO_READLINE=1 rlwrap -ppurple node"

To change both the prompt and the color, I’d use the following:

env NODE_NO_READLINE=1 rlwrap -ppurple -S "::>" node

Now my prompt would be:

::>

in purple.

The especially useful component of rlwrap is its ability to persist history across REPL sessions. By default, we have access to command-line history only within a REPL session. By using rlwrap, the next time we access REPL, not only will we have access to a history of commands within the current session, but also a history of commands in past sessions (and other command-line entries). In the following session output, the commands shown were not typed in, but were instead pulled from history with the up arrow key:

# env NODE_NO_READLINE=1 rlwrap -ppurple -S "::>" node
::>e = ['a','b'];
[ 'a', 'b' ]
::>3 > 2 > 1;
false

As helpful as rlwrap is, we still end up with undefined every time we type in an expression that doesn’t return a value. However, we can adjust this, and other functionality, just by creating our own custom REPL, discussed next.

Node provides us access to creating our own custom REPL. To do so, first we need to include the REPL module (repl):

var repl = require("repl");

To create a new REPL, we call the start method on the repl object. The syntax for this method is:

repl.start([prompt], [stream], [eval], [useGlobal], [ignoreUndefined]);

All of the parameters are optional. If not provided, default values will be used for each as follows:

I find the undefined expression result in REPL to be unedifying, so I created my own REPL. It took exactly two lines of code (not including the comment):

repl = require("repl");

// start REPL with ignoreUndefined set to true
repl.start("node via stdin> ", null, null, null, true);

I ran the file, repl.js, using Node:

node repl.js

Then I used the custom REPL just like I use the built-in version, except now I have a different prompt and no longer get the annoying undefined after the first variable assignment. I do still get the other responses that aren’t undefined:

node via stdin> var ct = 0;
node via stdin> ct++;
0
node via stdin> console.log(ct);
1
node via stdin> ++ct;
2
node via stdin> console.log(ct);
2

In my code, I wanted the defaults for all but prompt and ignoreUndefined. Setting the other parameters to null triggers Node to use the default values for each.

You can replace the eval function with your custom REPL. The only requirement is that it has a specific format:

function eval(cmd, callback) {
   callback(null, result);
}

The stream option is interesting. You can run multiple versions of REPL, taking input from both the standard input (the default), as well as sockets. The documentation for REPL at the Node.js site provides an example of a REPL listening in on a TCP socket, using code similar to the following:

var repl = require("repl"),
    net = require("net");

// start REPL with ignoreUndefined set to true
repl.start("node via stdin> ", null, null, null, true);

net.createServer(function (socket) {
  repl.start("node via TCP socket> ", socket);

}).listen(8124);

When you run the application, you get the standard input prompt where the Node application is running. However, you can also access REPL via TCP. I used PuTTY as a Telnet client to access this TCP-enabled version of REPL. It does work...to a point. I had to issue a .clear first, the formatting is off, and when I tried to use the underscore to reference the last expression, Node didn’t know what I was talking about, as shown in Figure 2-1.

Figure 2-1. PuTTY and REPL via TCP don’t exactly like each other

I also tried with the Windows 7 Telnet client, and the response was even worse. However, using my Linux Telnet client worked without a hitch.

The problem here, as you might expect, is Telnet client settings. However, I didn’t pursue it further, because running REPL from an exposed Telnet socket is not something I plan to implement, and not something I would recommend, either—at least, not without heavy security. It’s like using eval() in your client-side code, and not scrubbing the text your users send you to run—but worse.

You could keep a running REPL and communicate via a Unix socket with something like the GNU Netcat utility:

nc -U /tmp/node-repl-sock

You can type in commands no differently than typing them in using stdin. Be aware, though, if you’re using either a TCP or Unix socket, that any console.log commands are printed out to the server console, not to the client:

console.log(someVariable); // actually printed out to server

An application option that I consider to be more useful is to create a REPL application that preloads modules. In the application in Example 2-1, after the REPL is started, the http, os, and util modules are loaded and assigned to context properties.

var repl = require('repl');
var context = repl.start(">>", null, null, null, true).context;

// preload in modules
context.http = require('http');
context.util = require('util');
context.os = require('os');

Running the application with Node brings up the REPL prompt, where we can then access the modules:

>>os.hostname();
'einstein'
>>util.log('message');
5 Feb 11:33:15 - message
>>

If you want to run the REPL application like an executable in Linux, add the following line as the first line in the application:

#!/usr/local/bin/node

Modify the file to be an executable and run it:

# chmod u+x replcontext.js
# ./replcontext.js
>>

global is the global namespace object. In some ways, it’s similar to windows in a browser environment, in that it provides access to global properties and methods and doesn’t have to be explicitly referenced by name.

From REPL, you can print out the global object to the console like so:

> console.log(global)

What prints out is the interface for all of the other global objects, as well as a good deal of information about the system in which you’re running.

I mentioned that global is like the windows object in a browser, but there are key differences—and not just the methods and properties available. The windows object in a browser is truly global in nature. If you define a global variable in client-side JavaScript, it’s accessible by the web page and by every single library. However, if you create a variable at the top-level scope in a Node module (a variable outside a function), it only becomes global to the module, not to all of the modules.

You can actually see what happens to the global object when you define a module/global variable in REPL. First, define the top-level variable:

> var test = "This really isn't global, as we know global";

Then print out global:

> console.log(global);

You should see your variable, as a new property of global, at the bottom. For another interesting perspective, assign global to a variable, but don’t use the var keyword:

gl = global;

The global object interface is printed out to the console, and at the bottom you’ll see the local variable assigned as a circular reference:

> gl = global;
...
  gl: [Circular],
  _: [Circular] }

Any other global object or method, including require, is part of the global object’s interface.

When Node developers discuss context, they’re really referring to the global object. In Example 2-1 in Chapter 2, the code accessed the context object when creating a custom REPL object. The context object is a global object. When an application creates a custom REPL, it exists within a new context, which in this case means it has its own global object. The way to override this and use the existing global object is to create a custom REPL and set the useGlobal flag to true, rather than the default of false.

Modules exist in their own global namespace, which means that if you define a top-level variable in one module, it is not available in other modules. More importantly, it means that only what is explicitly exported from the module becomes part of whatever application includes the module. In fact, you can’t access a top-level module variable in an application or other module, even if you deliberately try.

To demonstrate, the following code contains a very simple module that has a top-level variable named globalValue, and functions to set and return the value. In the function that returns the value, the global object is printed out using a console.log method call.

var globalValue;

exports.setGlobal = function(val) {
   globalValue = val;
};

exports.returnGlobal = function() {
   console.log(global);
   return globalValue;
};

We might expect that in the printout of the global object we’ll see globalValue, as we do when we set a variable in our applications. This doesn’t happen, though.

Start a REPL session and issue a require call to include the new module:

> var mod1 = require('./mod1.js');

Set the value and then ask for the value back:

> mod1.setGlobal(34);
> var val = mod1.returnGlobal();

The console.log method prints out the global object before returning its globally defined value. We can see at the bottom the new variable holding a reference to the imported module, but val is undefined because the variable hasn’t yet been set. In addition, the output includes no reference to that module’s own top-level globalValue:

  mod1: { setGlobal: [Function], returnGlobal: [Function] },
  _: undefined,
  val: undefined }

If we ran the command again, then the outer application variable would be set, but we still wouldn’t see globalValue:

  mod1: { setGlobal: [Function], returnGlobal: [Function] },
  _: undefined,
  val: 34 }

The only access we have to the module data is by whatever means the module provides. For JavaScript developers, this means no more unexpected and harmful data collisions because of accidental or intentional global variables in libraries.

Each Node application is an instance of a Node process object, and as such, comes with certain built-in functionality.

Many of the process object’s methods and properties provide identification or information about the application and its environment. The process.execPath method returns the execution path for the Node application; process.version provides the Node version; and process.platform identifies the server platform:

console.log(process.execPath);
console.log(process.version);
console.log(process.platform);

This code returns the following in my system (at the time of this writing):

/usr/local/bin/node
v0.6.9
linux

The process object also wraps the STDIO streams stdin, stdout, and stderr. Both stdin and stdout are asynchronous, and are readable and writable, respectively. stderr, however, is a synchronous, blocking stream.

To demonstrate how to read and write data from stdin and stdout, in Example 3-1 the Node application listens for data in stdin, and repeats the data to stdout. The stdin stream is paused by default, so we have to issue a resume call before sending data.

process.stdin.resume();

process.stdin.on('data', function (chunk) {
  process.stdout.write('data: ' + chunk);
});

Run the application using Node, and then start typing into the terminal. Every time you type something and press Enter, what you typed is reflected back to you.

Another useful process method is memoryUsage, which tells us how much memory the Node application is using. This could be helpful for performance tuning, or just to satisfy your general curiosity about the application. The response has the following structure:

{ rss: 7450624, heapTotal: 2783520, heapUsed: 1375720 }

The heapTotal and heapUsed properties refer to the V8 engine’s memory usage.

The last process method I’m going to cover is process.nextTick. This method attaches a callback function that’s fired during the next tick (loop) in the Node event loop.

You would use process.nextTick if you wanted to delay a function for some reason, but you wanted to delay it asynchronously. A good example would be if you’re creating a new function that has a callback function as a parameter and you want to ensure that the callback is truly asynchronous. The following code is a demonstration:

function asynchFunction = function (data, callback) {
   process.nextTick(function() {
      callback(val);
   });
);

If we just called the callback function, then the action would be synchronous. Now, the callback function won’t be called until the next tick in the event loop, rather than right away.

You could use setTimeout with a zero (0) millisecond delay instead of process.nextTick:

setTimeout(function() {
   callback(val);
}, 0);

However, setTimeout isn’t as efficient as process.nextTick. When they were tested against each other, process.nextTick was called far more quickly than setTimeout with a zero-millisecond delay. You might also use process.nextTick if you’re running an application that has a function performing some computationally complex, and time-consuming, operation. You could break the process into sections, each called via process.nextTick, to allow other requests to the Node application to be processed without waiting for the time-consuming process to finish.

Of course, the converse of this is that you don’t want to break up a process that you need to ensure executes sequentially, because you may end up with unexpected results.

The Buffer class, also a global object, is a way of handling binary data in Node. In the section Servers, Streams, and Sockets later in the chapter, I’ll cover the fact that streams are often binary data rather than strings. To convert the binary data to a string, the data encoding for the stream socket is changed using setEncoding.

As a demonstration, you can create a new buffer with the following:

var buf = new Buffer(string);

If the buffer holds a string, you can pass in an optional second parameter with the encoding. Possible encodings are:

You can also write a string to an existing buffer, providing an optional offset, length, and encoding:

buf.write(string); // offset defaults to 0, length defaults to
                     buffer.length - offset, encoding is utf8

Data sent between sockets is transmitted as a buffer (in binary format) by default. To send a string instead, you either need to call setEncoding directly on the socket, or specify the encoding in the function that writes to the socket. By default, the TCP (Transmission Control Protocol) socket.write method sets the second parameter to utf8, but the socket returned in the connectionListener callback to the TCP createServer function sends the data as a buffer, not a string.

We can create a basic TCP server and client with the Node Net module. TCP forms the basis for most Internet applications, such as web service and email. It provides a way of reliably transmitting data between client and server sockets.

Creating the TCP server is a little different than creating the HTTP server in Example 1-1 in Chapter 1. We create the server, passing in a callback function. The TCP server differs from the HTTP server in that, rather than passing a requestListener, the TCP callback function’s sole argument is an instance of a socket listening for incoming connections.

Example 3-2 contains the code to create a TCP server. Once the server socket is created, it listens for two events: when data is received, and when the client closes the connection.

var net = require('net');

var server = net.createServer(function(conn) {
   console.log('connected');

   conn.on('data', function (data) {
      console.log(data + ' from ' + conn.remoteAddress + ' ' +
        conn.remotePort);
      conn.write('Repeating: ' + data);
   });

   conn.on('close', function() {
        console.log('client closed connection');
   });

}).listen(8124);

console.log('listening on port 8124');

There is an optional parameter for createServer: allowHalfOpen. Setting this parameter to true instructs the socket not to send a FIN when it receives a FIN packet from the client. Doing this keeps the socket open for writing (not reading). To close the socket, you’d then need to explicitly use the end method. By default, allowHalfOpen is false.

Notice how a callback function is attached to the two events via the on method. Many objects in Node that emit events provide a way to attach a function as an event listener by using the on method. This method takes the name of the event as first parameter, and the function listener as the second.

The TCP client is just as simple to create as the server, as shown in Example 3-3. The call to the setEncoding method on the client changes the encoding for the received data. As discussed earlier in the section Buffer, data is transmitted as a buffer, but we can use setEncoding to read it as a utf8 string. The socket’s write method is used to transmit the data. It also attaches listener functions to two events: data, for received data, and close, in case the server closes the connection.

 var net = require('net');

var client = new net.Socket();
client.setEncoding('utf8');

// connect to server
client.connect ('8124','localhost', function () {
    console.log('connected to server');
    client.write('Who needs a browser to communicate?');
});

// prepare for input from terminal
process.stdin.resume();

// when receive data, send to server
process.stdin.on('data', function (data) {
   client.write(data);
});

// when receive data back, print to console
client.on('data',function(data) {
    console.log(data);
});

// when server closed
client.on('close',function() {
    console.log('connection is closed');
});

The data being transmitted between the two sockets is typed in at the terminal, and transmitted when you press Enter. The client application first sends the string you just typed, which the TCP server writes out to the console. The server repeats the message back to the client, which in turn writes the message out to the console. The server also prints out the IP address and port for the client using the socket’s remoteAddress and remotePort properties. Following is the console output for the server after several strings were sent from the client (with the IP address edited out for security):

Hey, hey, hey, hey-now.
 from #ipaddress 57251
Don't be mean, we don't have to be mean.
 from #ipaddress 57251
Cuz remember, no matter where you go,
 from #ipaddress 57251
there you are.
 from #ipaddress 57251

The connection between the client and server is maintained until you kill one or the other using Ctrl-C. Whichever socket is still open receives a close event that’s printed out to the console. The server can also serve more than one connection from more than one client, since all the relevant functions are asynchronous.

As I mentioned earlier, TCP is the underlying transport mechanism for much of the functionality we use on the Internet today, including HTTP, which we’ll cover next.

You had a chance to work with the HTTP module in Chapter 1. We created servers using the createServer method, passing in the function that will act as the requestListener. Requests are processed as they come, asynchronously.

In a network, TCP is the transportation layer and HTTP is the application layer. If you scratch around in the modules included with Node, you’ll see that when you create an HTTP server, you’re inheriting functionality from the TCP-based net.Server.

For the HTTP server, the requestListener is a socket, while the http.ServerRequest object is a readable stream and the http.ServerResponse is a writable stream. HTTP adds another level of complexity because of the chunked transfer encoding it supports. The chunked transfer encoding allows transfer of data when the exact size of the response isn’t known until it’s fully processed. Instead, a zero-sized chunk is sent to indicate the end of a query. This type of encoding is useful when you’re processing a request such as a large database query output to an HTML table: writing the data can begin before the rest of the query data has been received.

The TCP examples earlier in this chapter, and the HTTP examples in Chapter 1, were both coded to work with network sockets. However, all of the server/socket modules can also connect to a Unix socket, rather than a specific network port. Unlike a network socket, a Unix or IPC (interprocess communication) socket enables communication between processes within the same system.

To demonstrate Unix socket communication, I duplicated Example 1-3’s code, but instead of binding to a port, the new server binds to a Unix socket, as shown in Example 3-4. The application also makes use of readFileSync, the synchronous version of the function to open a file and read its contents.

// create server
// and callback function
var http = require('http');
var fs = require('fs');

http.createServer(function (req, res) {

  var query = require('url').parse(req.url).query;
  console.log(query);
  file = require('querystring').parse(query).file;

  // content header
  res.writeHead(200, {'Content-Type': 'text/plain'});

  // increment global, write to client
  for (var i = 0; i<100; i++) {
    res.write(i + '\n');
  }

  // open and read in file contents
  var data = fs.readFileSync(file, 'utf8');
  res.write(data);
  res.end();
}).listen('/tmp/node-server-sock');

The client is based on a code sample provided in the Node core documentation for the http.request object at the Node.js site. The http.request object, by default, makes use of http.globalAgent, which supports pooled sockets. The size of this pool is five sockets by default, but you can adjust it by changing the agent.maxSockets value.

The client accepts the chunked data returned from the server, printing out to the console. It also triggers a response on the server with a couple of minor writes, as shown in Example 3-5.

var http = require('http');

var options = {
   method: 'GET',
   socketPath: '/tmp/node-server-sock',
   path: "/?file=main.txt"
};

var req = http.request(options, function(res) {
  console.log('STATUS: ' + res.statusCode);
  console.log('HEADERS: ' + JSON.stringify(res.headers));
  res.setEncoding('utf8');
  res.on('data', function (chunk) {
    console.log('chunk o\' data: ' + chunk);
  });
});

req.on('error', function(e) {
  console.log('problem with request: ' + e.message);
});

// write data to request body
req.write('data\n');
req.write('data\n');
req.end();

I didn’t use the asynchronous file read function with the http.request object because the connection is already closed when the asynchronous function is called and no file contents are returned.

Before leaving this section on the HTTP module, be aware that much of the behavior you’ve come to expect with Apache or other web servers isn’t built into a Node HTTP server. For instance, if you password-protect your website, Apache will pop up a window asking for your username and password; a Node HTTP server will not. If you want this functionality, you’re going to have to code for it.

TCP requires a dedicated connection between the two endpoints of the communication. UDP is a connectionless protocol, which means there’s no guarantee of a connection between the two endpoints. For this reason, UDP is less reliable and robust than TCP. On the other hand, UDP is generally faster than TCP, which makes it more popular for real-time uses, as well as technologies such as VoIP (Voice over Internet Protocol), where the TCP connection requirements could adversely impact the quality of the signal.

Node core supports both types of sockets. In the last couple of sections, I demonstrated the TCP functionality. Now, it’s UDP’s turn.

The UDP module identifier is dgram:

require ('dgram');

To create a UDP socket, use the createSocket method, passing in the type of socket—either udp4 or udp6. You can also pass in a callback function to listen for events. Unlike messages sent with TCP, messages sent using UDP must be sent as buffers, not strings.

Example 3-6 contains the code for a demonstration UDP client. In it, data is accessed via process.stdin, and then sent, as is, via the UDP socket. Note that we don’t have to set the encoding for the string, since the UDP socket accepts only a buffer, and the process.stdin data is a buffer. We do, however, have to convert the buffer to a string, using the buffer’s toString method, in order to get a meaningful string for the console.log method call that echoes the input.

var dgram = require('dgram');

var client = dgram.createSocket("udp4");

// prepare for input from terminal
process.stdin.resume();

process.stdin.on('data', function (data) {
   console.log(data.toString('utf8'));
   client.send(data, 0, data.length, 8124, "examples.burningbird.net",
      function (err, bytes) {
        if (err)
          console.log('error: ' + err);
        else
          console.log('successful');
   });
});

The UDP server, shown in Example 3-7, is even simpler than the client. All the server application does is create the socket, bind it to a specific port (8124), and listen for the message event. When a message arrives, the application prints it out using console.log, along with the IP address and port of the sender. Note especially that no encoding is necessary to print out the message—it’s automatically converted from a buffer to a string.

We didn’t have to bind the socket to a port. However, without the binding, the socket would attempt to listen in on every port.

var dgram = require('dgram');

var server = dgram.createSocket("udp4");

server.on ("message", function(msg, rinfo) {
   console.log("Message: " + msg + " from " + rinfo.address + ":"
                + rinfo.port);
});

server.bind(8124);

I didn’t call the close method on either the client or the server after sending/receiving the message. However, no connection is being maintained between the client and server—just the sockets capable of sending a message and receiving communication.

The communication stream between the sockets discussed in the previous sections is an implementation of the underlying abstract stream interface. Streams can be readable, writable, or both, and all streams are instances of EventEmitter, discussed in the upcoming section Events and EventEmitter.

It’s important to take away from this section that all of these communication streams, including process.stdin and process.stdout, are implementations of the abstract stream interface. Because of this underlying interface, there is basic functionality available in all streams in Node:

The last capability is one we haven’t covered yet. A simple way to demonstrate a pipe is to open a REPL session and type in the following:

> process.stdin.resume();
> process.stdin.pipe(process.stdout);

...and then enjoy the fact that everything you type from that point on is echoed back to you.

If you want to keep the output stream open for continued data, pass an option, { end: false }, to the output stream:

process.stdin.pipe(process.stdout, { end : false });

There is one additional object that provides a specific functionality to readable streams: readline. You include the Readline module with code like the following:

var readline = require('readline');

The Readline module allows line-by-line reading of a stream. Be aware, though, that once you include this module, the Node program doesn’t terminate until you close the interface and the stdin stream. The Node site documentation contains an example of how to begin and terminate a Readline interface, which I adapted in Example 3-8. The application asks a question as soon as you run it, and then outputs the answer. It also listens for any “command,” which is really any line that terminates with \n. If the command is .leave, it leaves the application; otherwise, it just repeats the command and prompts the user for more. A Ctrl-C or Ctrl-D key combination also causes the application to terminate.

var readline = require('readline');

// create a new interface
var interface = readline.createInterface(process.stdin, process.stdout, null);

// ask question
interface.question(">>What is the meaning of life?  ", function(answer) {
   console.log("About the meaning of life, you said " + answer);
   interface.setPrompt(">>");
   interface.prompt();
});

// function to close interface
function closeInterface() {
   console.log('Leaving interface...');
   process.exit();
}
// listen for .leave
interface.on('line', function(cmd) {
   if (cmd.trim() == '.leave') {
      closeInterface();
      return;
   } else {
      console.log("repeating command: " + cmd);
   }
   interface.setPrompt(">>");
   interface.prompt();
});

interface.on('close', function() {
    closeInterface();
});

Here’s an example session:

>>What is the meaning of life?  ===
About the meaning of life, you said ===
>>This could be a command
repeating command: This could be a command
>>We could add eval in here and actually run this thing
repeating command: We could add eval in here and actually run this thing
>>And now you know where REPL comes from
repeating command: And now you know where REPL comes from
>>And that using rlwrap replaces this Readline functionality
repeating command: And that using rlwrap replaces this Readline functionality
>>Time to go
repeating command: Time to go
>>.leave
Leaving interface...

This should look familiar. Remember from Chapter 2 that we can use rlwrap to override the command-line functionality for REPL. We use the following to trigger its use:

env NODE_NO_READLINE=1 rlwrap node

And now we know what the flag is triggering—it’s instructing REPL not to use Node’s Readline module for command-line processing, but to use rlwrap instead.

This is a quick introduction to the Node stream modules. Now it’s time to change course, and check out Node’s child processes.

There are four different techniques you can use to create a child process. The most common one is using the spawn method. This launches a command in a new process, passing in any arguments. In the following, we create a child process to call the Unix pwd command to print the current directory. The command takes no arguments:

var spawn = require('child_process').spawn,
    pwd = spawn('pwd');

pwd.stdout.on('data', function (data) {
  console.log('stdout: ' + data);
});

pwd.stderr.on('data', function (data) {
  console.log('stderr: ' + data);
});

pwd.on('exit', function (code) {
  console.log('child process exited with code ' + code);
});

Notice the events that are captured on the child process’s stdout and stderr. If no error occurs, any output from the command is transmitted to the child process’s stdout, triggering a data event on the process. If an error occurs, such as in the following where we’re passing an invalid option to the command:

var spawn = require('child_process').spawn,
    pwd = spawn('pwd', ['-g']);

Then the error gets sent to stderr, which prints out the error to the console:

stderr: pwd: invalid option -- 'g'
Try `pwd --help' for more information.

child process exited with code 1

The process exited with a code of 1, which signifies that an error occurred. The exit code varies depending on the operating system and error. When no error occurs, the child process exits with a code of 0.

The earlier code demonstrated sending output to the child process’s stdout and stderr, but what about stdin? The Node documentation for child processes includes an example of directing data to stdin. It’s used to emulate a Unix pipe (|) whereby the result of one command is immediately directed as input to another command. I adapted the example in order to demonstrate one of my favorite uses of the Unix pipe—being able to look through all subdirectories, starting in the local directory, for a file with a specific word (in this case, test) in its name:

find . -ls | grep test

Example 3-9 implements this functionality as child processes. Note that the first command, which performs the find, takes two arguments, while the second one takes just one: a term passed in via user input from stdin. Also note that, unlike the example in the Node documentation, the grep child process’s stdout encoding is changed via setEncoding. Otherwise, when the data is printed out, it would be printed out as a buffer.

var spawn = require('child_process').spawn,
    find = spawn('find',['.','-ls']),
    grep = spawn('grep',['test']);

grep.stdout.setEncoding('utf8');

// direct results of find to grep
find.stdout.on('data', function(data) {
   grep.stdin.write(data);
});

// now run grep and output results
grep.stdout.on('data', function (data) {
  console.log(data);
});

// error handling for both
find.stderr.on('data', function (data) {
  console.log('grep stderr: ' + data);
});
grep.stderr.on('data', function (data) {
  console.log('grep stderr: ' + data);
});

// and exit handling for both
find.on('exit', function (code) {
  if (code !== 0) {
    console.log('find process exited with code ' + code);
  }

  // go ahead and end grep process
  grep.stdin.end();
});

grep.on('exit', function (code) {
  if (code !== 0) {
    console.log('grep process exited with code ' + code);
  }
});

When you run the application, you’ll get a listing of all files in the current directory and any subdirectories that contain test in their filename.

All of the example applications up to this point work the same in Node 0.8 as in Node 0.6. Example 3-9 is an exception because of a change in the underlying API.

In Node 0.6, the exit event would not be emitted until the child process exits and all STDIO pipes are closed. In Node 0.8, the event is emitted as soon as the child process finishes. This causes the application to crash, because the grep child process’s STDIO pipe is closed when it tries to process its data. For the application to work in Node 0.8, the application needs to listen for the close event on the find child process, rather than the exit event:

// and exit handling for both
find.on('close', function (code) {
  if (code !== 0) {
    console.log('find process exited with code ' + code);
  }

  // go ahead and end grep process
  grep.stdin.end();
});

In Node 0.8, the close event is emitted when the child process exits and all STDIO pipes are closed.

In addition to spawning a child process, you can also use child_process.exec and child_process.execFile to run a command in a shell and buffer the results. The only difference between child_process.exec and child_process.execFile is that execFile runs an application in a file, rather than running a command.

The first parameter in the two methods is either the command or the file and its location; the second parameter is options for the command; and the third is a callback function. The callback function takes three arguments: error, stdout, and stderr. The data is buffered to stdout if no error occurs.

If the executable file contains:

#!/usr/local/bin/node

console.log(global);

the following application prints out the buffered results:

var execfile = require('child_process').execFile,
    child;

child = execfile('./app.js', function(error, stdout, stderr) {
  if (error == null) {
    console.log('stdout: ' + stdout);
  }
});

The last child process method is child_process.fork. This variation of spawn is for spawning Node processes.

What sets the child_process.fork process apart from the others is that there’s an actual communication channel established to the child process. Note, though, that each process requires a whole new instance of V8, which takes both time and memory.

Earlier I warned you that child processes that invoke Unix system commands won’t work with Windows, and vice versa. I know this sounds obvious, but not everyone knows that, unlike with JavaScript in browsers, Node applications can behave differently in different environments.

It wasn’t until recently that the Windows binary installation of Node even provided access to child processes. You also need to invoke whatever command you want to run via the Windows command interpreter, cmd.exe.

Example 3-10 demonstrates running a Windows command. In the application, Windows cmd.exe is used to create a directory listing, which is then printed out to the console via the data event handler.

var cmd = require('child_process').spawn('cmd', ['/c', 'dir\n']);

cmd.stdout.on('data', function (data) {
    console.log('stdout: ' + data);
});

cmd.stderr.on('data', function (data) {
    console.log('stderr: ' + data);
});

cmd.on('exit', function (code) {
    console.log('child process exited with code ' + code);
});

The /c flag passed as the first argument to cmd.exe instructs it to carry out the command and then terminate. The application doesn’t work without this flag. You especially don’t want to pass in the /K flag, which tells cmd.exe to execute the application and then remain because your application won’t terminate.

Colors is one of the simpler modules. You can use it to provide different color and style effects to the console.log output, and that’s it. However, it’s also a good demonstration of an effective module because it is simple to use, focuses on providing one service, and does a good job with that one service.

Testing how a module works is a great reason for using REPL. To try Colors, install it using npm like so:

$ npm install colors

Open a new REPL session and include the colors library:

> var colors = require('colors');

Because the Colors module is included in the current location’s node_modules subdirectory, Node is able to find it rather quickly.

Now try something out, such as the following:

console.log('This Node kicks it!'.rainbow.underline);

The result is a colorful, underlined rendering of your message. The style applies only for the one message—you’ll need to apply another style for another message.

If you’ve worked with jQuery, you recognize the chaining used to combine effects. The example makes use of two: a font effect, underlined, and a font color, rainbow.

Try another, this time zebra and bold:

console.log('We be Nodin'.zebra.bold);

You can change the style for sections of the console message:

console.log('rainbow'.rainbow, 'zebra'.zebra);

Why would something like Colors be useful? One reason is that it enables us to specify formatting for various events, such as displaying one color for errors in one module, another color or effect for warnings in a second module, and so on. To do this, you can use the Colors presets or create a custom theme:

> colors.setTheme({
....... mod1_warn: 'cyan',
....... mod1_error: 'red',
....... mod2_note: 'yellow'
....... });
> console.log("This is a helpful message".mod2_note);
This is a helpful message
> console.log("This is a bad message".mod1_error);
This is a bad message

Optimist is another module focused on solving a specific problem: parsing command options. That’s it, that’s all it does—but it does it very well.

As an example, this simple application uses the Optimist module to print command-line options to the console.

#!/usr/local/bin/node
var argv = require('optimist').argv;
console.log(argv.o + " " + argv.t);

You can run the application with short options. The following prints the values of 1 and 2 out to the console:

./app.js -o 1 -t 2

You can also process long options:

#!/usr/local/bin/node
var argv = require('optimist').argv;
console.log(argv.one + " " + argv.two);

and test with the following, resulting in a printout of My Name:

./app2.js --one="My" --two="Name"

You can also use the Optimist module to process Boolean and unhyphenated options.

node appname.js

#!/usr/local/bin/node

chmod a+x appname.js

./appname.js

Install the Underscore module with:

npm install underscore

According to the developers, Underscore is a utility-belt library for Node. It provides a lot of extended JavaScript functionality we’re used to with third-party libraries, such as jQuery or Prototype.js.

Underscore is so named because, traditionally, its functionality is accessed with an underscore (_), similar to jQuery’s $. Here’s an example:

var _ = require('underscore');
_.each(['apple','cherry'], function (fruit) { console.log(fruit); });

Of course, the problem with the underscore is that this character has a specific meaning in REPL. No worries, though—we can just use another variable, us:

var us = require('underscore');
us.each(['apple','cherry'], function(fruit) { console.log(fruit); });

Underscore provides expanded functionality for arrays, collections, functions, objects, chaining, and general utility. Fortunately, there’s also excellent documentation for all of its functionality, so I’ll forgo detailing any of it here.

However, I do want to mention one nice capability: a controlled way to extend Underscore with your own utility functions, via the mixin function. We can quickly try this method, and the others, in a REPL session:

> var us = require('underscore');
undefined
> us.mixin({
... betterWithNode: function(str) {
..... return str + ' is better with Node';
..... }
... });
> console.log(us.betterWithNode('chocolate'));
chocolate is better with Node

Of course, it makes more sense to extend Underscore from a module that we can reuse in our applications, which leads us to our next topic—creating our own custom modules.

You can split your module into separate JavaScript files, all located within a directory. Node can load the directory contents, as long as you organize the contents in one of two ways.

The first way is to provide a JSON file named package.json with information about the directory. The structure can contain other information, but the only entries relevant to Node are:

{ "name" : "mylibrary",
  "main" : "./mymodule/mylibrary.js" }

The first property, name, is the name of the module. The second, main, indicates the entry point for the module.

The second way is to include either an index.js or index.node file in the directory to serve as the main module entry point.

Why would you provide a directory rather than just a single module? The most likely reason is that you’re making use of existing JavaScript libraries, and just providing a “wrapper” file that wraps the exposed functions with exports statements. Another reason is that your library is so large that you want to break it down to make it easier to modify.

Regardless of the reason, be aware that all of the exported objects must be in the one main file that Node loads.

If you want to make your package available to others, you can promote it on your website, but you’ll be missing out on a significant audience. When you’re ready to publish a module, you’re going to want to add it to the list of modules at the Node.js website, and you’ll also want to publish it to the npm registry.

Earlier I mentioned the package.json file. It’s actually based on the CommonJS module system recommendations, which you can find at http://wiki.commonjs.org/wiki/Packages/1.0#Package_Descriptor_File (though check to see if there’s a more up-to-date version).

Among the required fields for the package.json file are:

The other fields are optional. Still, that’s a lot of fields. Thankfully, npm makes it easier to create this file. If you type the following at the command line:

npm init

the tool will run through the required fields, prompting you for each. When it’s done, it generates a package.json file.

In Chapter 3, Example 3-13, I started an object called inputChecker that checks incoming data for commands and then processes the command. The example demonstrated how to incorporate EventEmitter. Now we’re going to modify this simple object to make it usable by other applications and modules.

First, we’ll create a subdirectory in node_modules and name it inputcheck, and then move the existing inputChecker code file to it. We need to rename the file to index.js. Next, we need to modify the code to pull out the part that implements the new object. We’ll save it for a future test file. The last modification we’ll do is add the exports object, resulting in the code shown in Example 4-1.

var util = require('util');
var eventEmitter = require('events').EventEmitter;
var fs = require('fs');

exports.inputChecker = inputChecker;

function inputChecker(name, file) {
   this.name = name;
   this.writeStream = fs.createWriteStream('./' + file + '.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});
};

util.inherits(inputChecker,eventEmitter);
inputChecker.prototype.check = function check(input) {
  var self = this;
  var command = input.toString().trim().substr(0,3);
  if (command == 'wr:') {
     self.emit('write',input.substr(3,input.length));
  } else if (command == 'en:') {
     self.emit('end');
  } else {
     self.emit('echo',input);
  }
};

We can’t export the object function directly, because util.inherits expects an object to exist in the file named inputChecker. We’re also modifying the inputChecker object’s prototype later in the file. We could have changed these code references to use exports.inputChecker, but that’s kludgy. It’s just as easy to assign the object in a separate statement.

To create the package.json file, I ran npm init and answered each of the prompts. The resulting file is shown in Example 4-2.

{
  "author": "Shelley Powers <shelleyp@burningbird.net> (http://burningbird.net)",
  "name": "inputcheck",
  "description": "Looks for commands within the string and implements the commands",
  "version": "0.0.1",
  "homepage": "http://inputcheck.burningbird.net",
  "repository": {
    "url": "
  },
  "main": "inputcheck.js",
  "engines": {
    "node": "~0.6.10"
  },
  "dependencies": {},
  "devDependencies": {},
  "optionalDependencies": {}
}

The npm init command doesn’t prompt for dependencies, so we need to add them directly to the file. However, the inputChecker module isn’t dependent on any external modules, so we can leave these fields blank in this case.

At this point, we can test the new module to make sure it actually works as a module. Example 4-3 is the portion of the previously existing inputChecker application that tested the new object, now pulled out into a separate test application.

var inputChecker = require('inputcheck').inputChecker;

// testing new object and event handling
var ic = new inputChecker('Shelley','output');

ic.on('write', function(data) {
   this.writeStream.write(data, 'utf8');
});

ic.addListener('echo', function( data) {
   console.log(this.name + ' wrote ' + data);
});

ic.on('end', function() {
   process.exit();
});

process.stdin.resume();
process.stdin.setEncoding('utf8');
process.stdin.on('data', function(input) {
    ic.check(input);
});

We can now move the test application into a new examples subdirectory within the module directory to be packaged with the module as an example. Good practice demands that we also provide a test directory with one or more testing applications, as well as a doc directory with documentation. For a module this small, a README file should be sufficient. Lastly, we create a gzipped tarball of the module.

Once we’ve provided all we need to provide, we can publish the module.

The folks who brought us npm also provide a really great source for Node developers: the Developer Guide. It outlines everything we need to know about how to publish our modules.

The Guide specifies some additional requirements for the package.json file. In addition to the fields already created, we also need to add in a directories field with a hash of folders, such as the previously mentioned test and doc:

    "directories" : {
        "doc" : ".",
        "test" : "test",
        "example" : "examples"
    }

Before publishing, the Guide recommends we test that the module can cleanly install. To test for this, type the following in the root directory for the module:

npm install . -g

At this point, we’ve tested the inputChecker module, modified the package.json package to add directories, and confirmed that the package successfully installs.

Next, we need to add ourselves as npm users if we haven’t done so already. We do this by typing:

npm adduser

and following the prompts to add a username, a password, and an email address.

There’s one last thing to do:

npm publish

We can provide the path to the tarball or the directory. As the Guide warns us, everything in the directory is exposed unless we use a .npmignore list in the package.json file to ignore material. It’s better, though, just to remove anything that’s not needed before publishing the module.

Once published—and once the source is also uploaded to GitHub (if that’s the repository you’re using)—the module is now officially ready for others to use. Promote the module on Twitter, Google+, Facebook, your website, and wherever else you think people would want to know about the module. This type of promotion isn’t bragging—it’s sharing.

Step is a focused utility module that enables simplified control flow for serial and parallel execution. It can be installed using npm as follows:

npm install step

The Step module exports exactly one object. To use the object for serial execution, wrap your asynchronous function calls within functions that are then passed as parameters to the object. For instance, in Example 5-7, Step is used to read the contents of a file, modify the contents, and write them back to the file.

var fs = require('fs'),
    Step = require('step');

try {

   Step (
      function readData() {
          fs.readFile('./data/data1.txt', 'utf8', this);
      },
      function modify(err, text) {
         if (err) throw err;
         return text.replace(/somecompany\.com/g,'burningbird.net');
      },
      function writeData(err, text) {
          if (err) throw err;
          fs.writeFile('./data/data1.txt', text, this);
      }
    );
} catch(err) {
   console.error(err);
}

The first function in the Step sequence, readData, reads a file’s contents into a string, which is then passed to a second function. The second function modifies the string using replacement, and the result is passed to a third function. In the third function, the modified string is written back to the original file.

In more detail, the first function wraps the asynchronous fs.readFile. However, rather than pass a callback function as the last parameter, the code passes the this context. When the function is finished, its data and any possible error are sent to the next function, modify. The modify function isn’t an asynchronous function, as all it’s doing is replacing one substring for another in the string. It doesn’t require the this context, and just returns the result at the end of the function.

The last function gets the newly modified string and writes it back to the original file. Again, since it’s an asynchronous function, it gets this in place of the callback function. If we didn’t include this as the last parameter to the final function, any errors that occur wouldn’t be thrown and caught in the outer loop. If the boogabooga subdirectory didn’t exist with the following modified code:

      function writeFile(err, text) {
          if (err) throw err;
          fs.writeFile('./boogabooga/data/data1.txt');
      }

we’d never know that the write failed.

Even though the second function isn’t asynchronous, every function but the first in Step requires the error object as the first parameter for consistency. It’s just null by default in a synchronous function.

Example 5-7 performs part of the functionality of the application in Example 5-6. Could it do the rest of the functionality, especially handling modification to multiple files? The answer is yes, and no. Yes, it can do the work, but only if we throw in some kludgy code.

In Example 5-8, I added in the readir asynchronous function to get a list of files in a given subdirectory. The array of files is processed with a forEach command, like in Example 5-6, but the end of the call to readFile isn’t a callback function or this. In Step, the call to create the group object signals to reserve a parameter for a group result; the call to the group object in the readFile asynchronous function results in each of the callbacks being called in turn, and the results being grouped into an array for the next function.

var fs = require('fs'),
    Step = require('step'),
    files,
    _dir = './data/';

try {

   Step (
      function readDir() {
          fs.readdir(_dir, this);
      },
      function readFile(err, results) {
          if (err) throw err;
          files = results;
          var group = this.group();
          results.forEach(function(name) {
             fs.readFile(_dir + name, 'utf8', group());
          });
      },
      function writeAll(err, data) {
         if (err) throw err;
         for (var i = 0; i < files.length; i++) {
           var adjdata = data[i].replace(/somecompany\.com/g,'burningbird.net');
           fs.writeFile(_dir + files[i], adjdata, 'utf8',this);
         }
      }
    );
} catch(err) {
   console.log(err);
}

To preserve the filenames, the readdir result is assigned to a global variable, files. In the last Step function, a regular for loop cycles through the data to modify it, and then cycles through the files variable to get the filename. Both the filename and modified data are used in the last asynchronous call to writeFile.

One other approach we could have used if we wanted to hardcode the change to each file is to use the Step parallel feature. Example 5-9 performs a readFile on a couple of different files, passing in this.parallel() as the last parameter. This results in a parameter being passed to the next function for each readFile in the first function. The parallel function call also has to be used in the writeFile function in the second function, to ensure that each callback is processed in turn.

var fs = require('fs'),
    Step = require('step'),
    files;

try {

   Step (
      function readFiles() {
         fs.readFile('./data/data1.txt', 'utf8',this.parallel());
         fs.readFile('./data/data2.txt', 'utf8',this.parallel());
         fs.readFile('./data/data3.txt', 'utf8',this.parallel());
      },
      function writeFiles(err, data1, data2, data3) {
         if (err) throw err;
         data1 = data1.replace(/somecompany\.com/g,'burningbird.net');
         data2 = data2.replace(/somecompany\.com/g,'burningbird.net');
         data3 = data3.replace(/somecompany\.com/g,'burningbird.net');

         fs.writeFile('./data/data1.txt', data1, 'utf8', this.parallel());
         fs.writeFile('./data/data2.txt', data2, 'utf8', this.parallel());
         fs.writeFile('./data/data3.txt', data3, 'utf8', this.parallel());
      }
    );
} catch(err) {
   console.log(err);
}

It works, but it’s clumsy. It would be better to reserve the use of the parallel functionality for a sequence of different asynchronous functions that can be implemented in parallel, and the data processed post-callback.

As for our earlier application, rather than trying to force Step into contortions to fit our use case, we can use another library that provides the additional flexibility we need: Async.

The Async module provides functionality for managing collections, such as its own variation of forEach, map, and filter. It also provides some utility functions, including ones for memoization. However, what we’re interested in here are its facilities for handling control flow.

Install Async using npm like so:

npm install async

As mentioned earlier, Async provides control flow capability for a variety of asynchronous patterns, including serial, parallel, and waterfall. Like Step, it gives us a tool to tame the wild nested callback pyramid, but its approach is quite different. For one, we don’t insert ourselves between each function and its callback. Instead, we incorporate the callback as part of the process.

As an example, we’ve already identified that the pattern of the earlier application matches with Async’s waterfall, so we’ll be using the async.waterfall method. In Example 5-10, I used async.waterfall to open and read a data file using fs.readFile, perform the synchronous string substitution, and then write the string back to the file using fs.writeFile. Pay particular attention to the callback function used with each step in the application.

var fs = require('fs'),
    async = require('async');

try {
   async.waterfall([
      function readData(callback) {
         fs.readFile('./data/data1.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      function modify(text, callback) {
         var adjdata=text.replace(/somecompany\.com/g,'burningbird.net');
         callback(null, adjdata);
      },
      function writeData(text, callback) {
          fs.writeFile('./data/data1.txt', text, function(err) {
             callback(err,text);
          });
      }
    ], function (err, result) {
         if (err) throw err;
         console.log(result);
    });
} catch(err) {
   console.log(err);
}

The async.waterfall method takes two parameters: an array of tasks and an optional final callback function. Each asynchronous task function is an element of the async.waterfall method array, and each function requires a callback as the last of its parameters. It is this callback function that allows us to chain the asynchronous callback results without having to physically nest the functions. However, as you can see in the code, each function’s callback is handled as we would normally handle it if we were using nested callbacks—other than the fact that we don’t have to test the errors in each function. The callbacks look for an error object as first parameter. If we pass an error object in the callback function, the process is ended at this point, and the final callback routine is called. The final callback is when we can test for an error, and throw the error to the outer exception handling block (or otherwise handle).

The readData function wraps our fs.readFile call, which checks for an error, first. If an error is found, it throws the error, ending the process. If not, it issues a call to the callback as its last operation. This is the trigger to tell Async to invoke the next function, passing any relevant data. The next function isn’t asynchronous, so it does its processing, passing null as the error object, and the modified data. The last function, writeData, calls the asynchronous writeFile, using the passed-in data from the previous callback and then testing for an error in its own callback routine.

The processing is very similar to what we had in Example 5-4, but without the nesting (and having to test for an error in each function). It may seem more complicated than what we had in Example 5-4, and I wouldn’t necessarily recommend its use for such simple nesting, but look what it can do with a more complex nested callback. Example 5-11 duplicates the exact functionality from Example 5-6, but without the callback nesting and excessive indenting.

var fs = require('fs'),
    async = require('async'),
    _dir = './data/';

var writeStream = fs.createWriteStream('./log.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});
try {
   async.waterfall([
      function readDir(callback) {
         fs.readdir(_dir, function(err, files) {
            callback(err,files);
         });
      },
      function loopFiles(files, callback) {
         files.forEach(function (name) {
            callback (null, name);
         });
      },
      function checkFile(file, callback) {
         fs.stat(_dir + file, function(err, stats) {
            callback(err, stats, file);
         });
      },
      function readData(stats, file, callback) {
         if (stats.isFile())
            fs.readFile(_dir + file, 'utf8', function(err, data){
              callback(err,file,data);
          });
      },
      function modify(file, text, callback) {
         var adjdata=text.replace(/somecompany\.com/g,'burningbird.net');
         callback(null, file, adjdata);
      },
      function writeData(file, text, callback) {
          fs.writeFile(_dir + file, text, function(err) {
             callback(err,file);
          });
      },
      function logChange(file, callback) {
          writeStream.write('changed ' + file + '\n', 'utf8', function(err) {
             callback(err, file);
         });
      }
    ], function (err, result) {
         if (err) throw err;
         console.log('modified ' + result);
    });
} catch(err) {
   console.log(err);
}

Every last bit of functionality is present from Example 5-6. The fs.readdir method is used to get an array of directory objects. The Node forEach method (not the Async forEach) is used to access each specific object. The fs.stats method is used to get the stats for each object. stats is used to check for files, and when a file is found, it’s opened and its data accessed. The data is then modified, and passed on to be written back to the file via fs.writeFile. The operation is logged in the logfile and also echoed to the console.

Note that there is more data passed in some of the callbacks. Most of the functions need the filename as well as the text, so this is passed in the last several methods. Any amount of data can be passed in the methods, as long as the first parameter is the error object (or null, if no error object) and the last parameter in each function is the callback function.

We don’t have to check for an error in each asynchronous task function either, because Async tests the error object in each callback, and stops processing and calls the final callback function if an error is found. And we don’t have to worry about using special processing when handling an array of items, as we did when we used Step earlier in the chapter.

The other Async control flow methods, such as async.parallel and async.serial, perform in a like manner, with an array of tasks as the first method parameter, and a final optional callback as the second. How they process the asynchronous tasks differs, though, as you would expect.

The async.parallel method calls all of the asynchronous functions at once, and when they are each finished, calls the optional final callback. Example 5-12 uses async.parallel to read in the contents of three files in parallel. However, rather than an array of functions, this example uses an alternative approach that Async supports: passing in an object with each asynchronous task listed as a property of the object. The results are then printed out to the console when all three tasks have finished.

var fs = require('fs'),
    async = require('async');

try {
   async.parallel({
      data1 : function (callback) {
         fs.readFile('./data/data1.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      data2 : function (callback) {
         fs.readFile('./data/data2.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },
      data3 : function readData3(callback) {
         fs.readFile('./data/data3.txt', 'utf8', function(err, data){
              callback(err,data);
          });
      },

    }, function (err, result) {
         if (err) throw err;
         console.log(result);
    });
} catch(err) {
   console.log(err);
}

The results are returned as an array of objects, with each result tied to each of the properties. If the three data files in the example had the following content:

the result of running Example 5-12 is:

{ data1: 'apples\n', data2: 'oranges\n', data3: 'peaches\n' }

I’ll leave the testing of the other Async control flow methods as a reader exercise. Just remember that when you’re working with the Async control flow methods, all you need is to pass a callback to each asynchronous task and to call this callback when you’re finished, passing in an error object (or null) and whatever data you need.

You can install Connect using npm:

npm install connect

Connect is, in actuality, a framework in which you can use one or more middleware applications. The documentation for Connect is sparse. However, it is relatively simple to use once you’ve seen a couple of working examples.

npm install -d

In Example 6-3, I created a simple server application using Connect, and using two of the middleware^[1] bundled with Connect: connect.logger and connect.favicon. The logger middleware logs all requests to a stream—in this case, the default STDIO.output stream—and the favicon middleware serves up the favicon.ico file. The application includes the middleware using the use method on the Connect request listener, which is then passed as a parameter to the HTTP object’s createServer method.

var connect = require('connect');
var http = require('http');

var app = connect()
   .use(connect.favicon())
   .use(connect.logger())
   .use(function(req,res) {
      res.end('Hello World\n');
   });

http.createServer(app).listen(8124);

You can use any number of middleware—either built in with Connect or provided by a third party—by just including additional use states.

Rather than create the Connect request listener first, we can also incorporate the Connect middleware directly into the createServer method, as shown in Example 6-4.

var connect = require('connect');
var http = require('http');

http.createServer(connect()
   .use(connect.favicon())
   .use(connect.logger())
   .use(function(req,res) {
      res.end('Hello World\n');
   })).listen(8124);

Connect comes bundled with at least 20 middleware. I’m not going to cover them all in this section, but I am going to demonstrate enough of them so that you have a good understanding of how they work together.

var connect = require('connect'),
    http = require('http'),
    __dirname = '/home/examples';

http.createServer(connect()
   .use(connect.logger())
   .use(connect.static(_dirname + '/public_html'), {redirect: true})
   ).listen(8124);

99.28.217.189 - - [Sat, 25 Feb 2012 02:18:22 GMT] "GET /example1.html HTTP/1.1" 304
- "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko)
 Chrome/17.0.963.56 Safari/535.11"
99.28.217.189 - - [Sat, 25 Feb 2012 02:18:22 GMT] "GET /phoenix5a.png HTTP/1.1" 304
 - "http://examples.burningbird.net:8124/example1.html"
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko)
Chrome/17.0.963.56 Safari/535.11"
99.28.217.189 - - [Sat, 25 Feb 2012 02:18:22 GMT] "GET /favicon.ico HTTP/1.1"
304 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko)
 Chrome/17.0.963.56 Safari/535.11"
99.28.217.189 - - [Sat, 25 Feb 2012 02:18:28 GMT]
"GET /html5media/chapter2/example16.html HTTP/1.1" 304 - "-"
"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko)
Chrome/17.0.963.56 Safari/535.11"

var connect = require('connect'),
    http = require('http'),
    fs = require('fs'),
    __dirname = '/home/examples';

var writeStream = fs.createWriteStream('./log.txt',
      {'flags' : 'a',
       'encoding' : 'utf8',
       'mode' : 0666});

http.createServer(connect()
   .use(connect.logger({format : 'dev', stream : writeStream }))
   .use(connect.static(__dirname + '/public_html'))
   ).listen(8124);

GET /example1.html 304 4ms
GET /phoenix5a.png 304 1ms
GET /favicon.ico 304 1ms
GET /html5media/chapter2/example16.html 304 2ms
GET /html5media/chapter2/bigbuckposter.jpg 304 1ms
GET /html5media/chapter2/butterfly.png 304 1ms
GET /html5media/chapter2/example1.html 304 1ms
GET /html5media/chapter2/bigbuckposter.png 304 0ms
GET /html5media/chapter2/videofile.mp4 304 0ms

connect.parseCookie and connect.cookieSession

The scratch file server didn’t provide any functionality to work with HTTP cookies, nor did it handle session state. Luckily for us, both are handled with Connect middleware.

Chances are, one of your first JavaScript client applications was to create an HTTP request cookie. The connect.parseCookie middleware provides the functionality that allows us to access the cookie data on the server. It parses the cookie header, populating req.cookies with the cookie/data pairs. Example 6-6 shows a simple web server that extracts the cookie for a key value of username and writes a somewhat creepy but relevant message to stdout.

Example 6-6. Accessing an HTTP request cookie, and using it for a console message

var connect = require('connect')
  , http = require('http');

var app = connect()
  .use(connect.logger('dev'))
  .use(connect.cookieParser())
  .use(function(req, res, next) {
     console.log('tracking ' + req.cookies.username);
     next();
   })
  .use(connect.static('/home/examples/public_html'));

http.createServer(app).listen(8124);
console.log('Server listening on port 8124');

I’ll get into the use of the anonymous function, and especially the purpose of next, in the section Custom Connect Middleware. Focusing for now on connect.cookieParser, we see that this middleware intercepts the incoming request, pulls the cookie data out of the header, and stores the data in the request object. The anonymous function then accesses the username data from the cookies object, outputting it to the console.

To create an HTTP response cookie, we pair connect.parseCookie with connect.cookieSession, which provides secure session persistence. Text is passed as a string to the connect.cookieParser function, providing a secret key for session data. The data is added directly to the session object. To clear the session data, set the session object to null.

Example 6-7 creates two functions—one to clear the session data, and one to output a tracking message—that are used as middleware for incoming requests. They’re added as middleware in addition to logger, parseCookie, cookieSession, and static. The user is prompted for his or her username in the client page, which is then used to set a request cookie. On the server, the username and the number of resources the person has accessed in the current session are persisted via an encrypted response cookie.

Example 6-7. Using a session cookie to track resource accesses

var connect = require('connect')
  , http = require('http');

// clear all session data
function clearSession(req, res, next) {
  if ('/clear' == req.url) {
     req.session = null;
     res.statusCode = 302;
     res.setHeader('Location', '/');
     res.end();
  } else {
     next();
  }
}

// track user
function trackUser(req, res, next) {
  req.session.ct = req.session.ct || 0;
  req.session.username = req.session.username || req.cookies.username;
  console.log(req.cookies.username + ' requested ' +
                       req.session.ct++ + ' resources this session');
  next();
}

// cookie and session
var app = connect()
  .use(connect.logger('dev'))
  .use(connect.cookieParser('mumble'))
  .use(connect.cookieSession({key : 'tracking'}))
  .use(clearSession)
  .use(trackUser);

// static server
app.use(connect.static('/home/examples/public_html'));
// start server and listen
http.createServer(app).listen(8124);
console.log('Server listening on port 8124');

Figure 6-2 shows a web page accessed through the server application in Example 6-8. The JavaScript console is open to display both cookies. Note that the response cookie, unlike the request, is encrypted.

Figure 6-2. JavaScript console open in Chrome, displaying request and response cookies

The number of documents the user accesses is tracked, either until the user accesses the /clear URL (in which case the session object is set to null) or closes the browser, ending the session.

Example 6-7 also made use of a couple of custom middleware functions. In the next (and final) section on Connect, we’ll discuss how these work with Connect, and how to create a third-party middleware.

In Example 6-7 in the previous section, we created two functions as Connect middleware in order to process incoming requests before the final static server. The three parameters passed to the functions are the HTTP request and response objects, and next, a callback function. These three form the signature for a Connect middleware function.

To get a closer look at how Connect middleware works, let’s examine one used in earlier code, connect.favicon. This is nothing more than a simple function to either serve the default favicon.ico or provide a custom path:

connect()
.use (connect.favicon('someotherloc/favicon.ico'))

The reason I cover connect.favicon, other than its usefulness, is that it’s one of the simplest middleware, and therefore easy to reverse engineer.

The source code for connect.favicon, especially when compared with other source codes, shows that all exported middleware return a function with the following minimum signature or profile:

return function(req, res, next)

The next callback, passed as the last parameter to the function, is called if the middleware does not process the current request, or doesn’t process it completely. The next callback is also called if the middleware has an error, and an error object is returned as the parameter, as shown in Example 6-8.

module.exports = function favicon(path, options){
  var options = options || {}
    , path = path || __dirname + '/../public/favicon.ico'
    , maxAge = options.maxAge || 86400000;

  return function favicon(req, res, next){
    if ('/favicon.ico' == req.url) {
      if (icon) {
        res.writeHead(200, icon.headers);
        res.end(icon.body);
      } else {
        fs.readFile(path, function(err, buf){
          if (err) return next(err);
          icon = {
            headers: {
                'Content-Type': 'image/x-icon'
              , 'Content-Length': buf.length
              , 'ETag': '"' + utils.md5(buf) + '"'
              , 'Cache-Control': 'public, max-age=' + (maxAge / 1000)
            },
            body: buf
          };
          res.writeHead(200, icon.headers);
          res.end(icon.body);
        });
      }
    } else {
      next();
    }
  };
};

The next callback is, of course, how the chained functions are called, in sequence. In an incoming request, if the middleware can completely handle the request, such as the request favicon.ico request, no further middleware are invoked. This is why you would include the connect.favicon middleware before connect.logger in your applications—to prevent requests for favicon.ico from cluttering up the logs:

http.createServer(connect()
   .use(connect.favicon('/public_html/favicon.ico'))
   .use(connect.logger())
   .use(connect.static(_dirname + '/public_html'))
   ).listen(8124);

You’ve seen how you can create a custom Connect middleware directly in the application, and how a bundled Connect middleware looks, but how would you create a third-party middleware that’s not going to be embedded directly in the application?

To create an external Connect middleware, create the module as you would any other module, but make sure it has all the pieces that Connect requires—specifying the three parameters (req, res, and next), and that it calls next if it doesn’t completely handle the request.

Example 6-9 creates a Connect middleware that checks to see if the requested file exists and that it is a file (not a directory). If the request is a directory, it returns a 403 status code and a custom message. If the file doesn’t exist, it returns a 404 status code and, again, a custom message. If neither happens, then it calls next to trigger the Connect middleware into invoking the next function (in this case, connect.static).

var fs = require('fs');

module.exports = function customHandler(path, missingmsg, directorymsg) {
   if (arguments.length < 3) throw new Error('missing parameter in customHandler');

   return function customHandler(req, res, next) {
      var pathname = path + req.url;
      console.log(pathname);
      fs.stat(pathname, function(err, stats) {
         if (err) {
            res.writeHead(404);
            res.write(missingmsg);
            res.end();
         } else if (!stats.isFile()) {
            res.writeHead(403);
            res.write(directorymsg);
            res.end();
         } else {
            next();
         }
      });
   }
}

The custom Connect middleware throws an error when one occurs, but if an error occurs within the returned function, next is called with an error object:

next(err);

The following code shows how we can use this custom middleware in an application:

var connect = require('connect'),
    http = require('http'),
    fs = require('fs'),
    custom = require('./custom'),
    base = '/home/examples/public_html';

http.createServer(connect()
   .use(connect.favicon(base + '/favicon.ico'))
   .use(connect.logger())
   .use(custom(base + '/public_html', '404 File Not Found',
          '403 Directory Access Forbidden'))
   .use(connect.static(base))
   ).listen(8124);

Connect does have an errorHandler function, but it doesn’t serve the purpose we’re trying to achieve. Rather, its purpose is to provide a formatted output of an exception. You’ll see it in use with an Express application in Chapter 7.

There are several other bundled middleware, as well as a significant number of third-party middleware you can use with Connect. In addition, Connect forms the middleware layer for the Express application framework, discussed in Chapter 7. First, though, let’s take a quick look at two other types of services necessary for many Node applications: routers and proxies.

The route, or route path, given in Example 7-1 is just a simple / (forward slash) signifying the root address. Express compiles all routes to a regular expression object internally, so you can use strings with special characters, or just use regular expressions directly in the path strings.

To demonstrate, I created a bare-bones routing path application in Example 7-2 that listens for three different routes. If a request is made to the server for one of these routes, the parameters from the request are returned to the sender using the Express response object’s send method.

var express = require('express')
  , http = require('http');

var app = express();

app.configure(function(){
});

app.get(/^\/node?(?:\/(\d+)(?:\.\.(\d+))?)?/, function(req, res){
   console.log(req.params);
   res.send(req.params);
});

app.get('/content/*',function(req,res) {
   res.send(req.params);
});

app.get("/products/:id/:operation?", function(req,res) {
   console.log(req);
   res.send(req.params.operation + ' ' + req.params.id);
});

http.createServer(app).listen(3000);

console.log("Express server listening on port 3000");

We’re not doing any routing to views or handling any static content, so we didn’t need to provide middleware in the app.configure method. However, we do need to call the app.configure method if we want to gracefully handle requests that don’t match any of the routes. The application is also using the default environment (development).

The first of the app.get method calls is using a regular expression for the path. This regular expression is adapted from one provided in the Express Guide, and listens for any request for a node. If the request also provides a unique identifier or range of identifiers, this is captured in the request object’s params array, which is sent back as a response. The following requests:

node
nodes
 /node/566
/node/1..10
/node/50..100/something

return the following params array values:

[null, null]
[null, null]
["566", null]
["1", "10"]
["50", "100"]

The regular expression is looking for a single identifier or a range of identifiers, given as two values with a range indicator (..) between them. Anything after the identifier or range is ignored. If no identifier or range is provided, the parameters are null.

The code to process the request doesn’t use the underlying HTTP response object’s write and end methods to send the parameters back to the requester; instead, it uses the Express send method. The send method determines the proper headers for the response (given the data type of what’s being sent) and then sends the content using the underlying HTTP end method.

The next app.get is using a string to define the routing path pattern. In this case, we’re looking for any content item. This pattern will match anything that begins with /content/. The following requests:

/content/156
/content/this_is_a_story
/content/apples/oranges

result in the following params values:

["156"]
["this_is_a_story"]
["apples/oranges"]

The asterisk (*) is liberal in what it accepts, and everything after content/ is returned.

The last app.get method is looking for a product request. If a product identifier is given, it can be accessed directly via params.id. If an operation is given, it can be accessed directly via params.operation. Any combination of the two values is allowed, except not providing at least one identifier or one operation.

The following URLs:

/products/laptopJK3444445/edit
/products/fordfocus/add
/products/add
/products/tablet89/delete
/products/

result in the following returned values:

edit laptopJK3444445
add fordfocus
undefined add
delete tablet89
Cannot GET /products/

The application outputs the request object to the console. When running the application, I directed the output to an output.txt file so I could examine the request object more closely:

node app.js > output.txt

The request object is a socket, of course, and we’ll recognize much of the object from our previous work exploring the Node HTTP request object. What we’re mainly interested in is the route object added via Express. Following is the output for the route object for one of the requests:

  route:
   { path: '/products/:id/:operation?',
     method: 'get',
     callbacks: [ [Function] ],
     keys: [ [Object], [Object] ],
     regexp: /^\/products\/(?:([^\/]+?))(?:\/([^\/]+?))?\/?$/i,
     params: [ id: 'laptopJK3444445', operation: 'edit' ] },

Note the generated regular expression object, which converts my use of the optional indicator (:) in the path string into something meaningful for the underlying JavaScript engine (thankfully, too, since I’m lousy at regular expressions).

Now that we have a better idea of how the routing paths work, let’s look more closely at the use of the HTTP verbs.

In the previous examples, we used app.get to process incoming requests. This method is based on the HTTP GET method and is useful if we’re looking for data, but not if we want to process new incoming data or edit or delete existing data. We need to make use of the other HTTP verbs to create an application that maintains as well as retrieves data. In other words, we need to make our application RESTful.

Let’s say our application is managing that most infamous of products, the widget. To create a new widget, we’ll need to create a web page providing a form that gets the information about the new widget. We can generate this form with the application, and I’ll demonstrate this approach in Chapter 8, but for now we’ll use a static web page, shown in Example 7-3.

<!doctype html>
<html lang="en">
<head>
 <meta charset="utf-8" />
 <title>Widgets</title>
</head>
<body>
<form method="POST" action="/widgets/add"
enctype="application/x-www-form-urlencoded">

 <p>Widget name: <input type="text" name="widgetname" id="widgetname"
size="25" required/></p>

 <p>Widget Price: <input type="text"
pattern="^\$?([0-9]{1,3},([0-9]{3},)*[0-9]{3}|[0-9]+)(.[0-9][0-9])?$"
name="widgetprice" id="widgetprice" size="25" required/></p>

 <p>Widget Description: <br /><textarea name="widgetdesc" id="widgetdesc"
cols="20" rows="5">No Description</textarea>
 <p>

 <input type="submit" name="submit" id="submit" value="Submit"/>
 <input type="reset" name="reset" id="reset" value="Reset"/>
 </p>
 </form>
</body>

The page takes advantage of the new HTML5 attributes required and pattern to provide validation of data. Of course, this works only with browsers that support HTML5, but for now, I’ll assume you’re using a modern HTML5-capable browser.

The widget form requires a widget name, price (with an associated regular expression to validate the data structure in the pattern attribute), and description. Browser-based validation should ensure we get the three values, and that the price is properly formatted as US currency.

In the Express application, we’re just going to persist new widgets in memory, as we want to focus purely on the Express technology at this time. As each new widget is posted to the application, it’s added to an array of widgets via the app.post method. Each widget can be accessed by its application-generated identifier via the app.get method. Example 7-4 shows the entire application.

var express = require('express')
  , http = require('http')
  , app = express();

app.configure(function(){
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.static(__dirname + '/public'));
  app.use(express.bodyParser());
  app.use(app.router);
});

app.configure('development', function(){
  app.use(express.errorHandler());
});

// in memory data store
var widgets = [
  { id : 1,
    name : 'My Special Widget',
    price : 100.00,
    descr : 'A widget beyond price'
  }
]

// add widget
app.post('/widgets/add', function(req, res) {
  var indx = widgets.length + 1;
  widgets[widgets.length] =
   { id : indx,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     descr : req.body.widgetdesc };
  console.log('added ' + widgets[indx-1]);
  res.send('Widget ' + req.body.widgetname + ' added with id ' + indx);
});

// show widget
app.get('/widgets/:id', function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   if (!widgets[indx])
      res.send('There is no widget with id of ' + req.params.id);
   else
      res.send(widgets[indx]);
});

http.createServer(app).listen(3000);

console.log("Express server listening on port 3000");

The first widget is seeded into the widget array, so we have existing data if we want to immediately query for a widget without adding one first. Note the conditional test in app.get to respond to a request for a nonexistent or removed widget.

Running the application (example4.js in the examples), and accessing the application using / or /index.html (or /example3.html, in the examples) serves up the static HTML page with the form. Submitting the form generates a page displaying a message about the widget being added, as well as its identifier. We can then use the identifier to display the widget—in effect, a dump of the widget object instance:

http://whateverdomain.com:3000/widgets/2

It works...but there’s a problem with this simple application.

First, it’s easy to make a typo in the widget fields. You can’t put in data formatted as anything other than currency in the price field, but you can put in the wrong price. You can also easily type in the wrong name or description. What we need is a way to update a widget, as well as a way to remove a widget we no longer need.

The application needs to incorporate support for two other RESTful verbs: PUT and DELETE. PUT is used to update the widget, while DELETE is used to remove it.

To update the widget, we’ll need a form that comes prepopulated with the widget data in order to edit it. To delete the widget, we’ll need a form that confirms whether we truly want to delete the widget. In an application, these are generated dynamically using a template, but for now, since we’re focusing on the HTTP verbs, I created static web pages that edit and then delete the already created widget, widget 1.

The form for updating widget 1 is shown in the following code. Other than being populated with the information for widget 1, there is just one other difference between this form and the form to add a new widget: the addition of a hidden field named _method, shown in bold text:

<form method="POST" action="/widgets/1/update"
enctype="application/x-www-form-urlencoded">

 <p>Widget name: <input type="text" name="widgetname"
 id="widgetname" size="25" value="My Special Widget" required/></p>

 <p>Widget Price: <input type="text"
 pattern="^\$?([0-9]{1,3},([0-9]{3},)*[0-9]{3}|[0-9]+)(.[0-9][0-9])?$"
 name="widgetprice" id="widgetprice" size="25" value="100.00" required/></p>

 <p>Widget Description: <br />
 <textarea name="widgetdesc" id="widgetdesc" cols="20"
 rows="5">A widget beyond price</textarea>
 <p>

 <input type="hidden" value="put" name="_method" />

 <input type="submit" name="submit" id="submit" value="Submit"/>
 <input type="reset" name="reset" id="reset" value="Reset"/>
 </p>
 </form>

Since PUT and DELETE are not supported in the form method attribute, we have to add them using a hidden field with a specific name, _method, and give them a value of either put, for PUT, or delete for DELETE.

The form to delete the widget is simple: it contains the hidden _method field, and a button to confirm the deletion of widget 1:

<p>Are you sure you want to delete Widget 1?</p>
<form method="POST" action="/widgets/1/delete"
 enctype="application/x-www-form-urlencoded">

 <input type="hidden" value="delete" name="_method" />

 <p>
 <input type="submit" name="submit" id="submit" value="Delete Widget 1"/>
 </p>
</form>

To ensure that the HTTP verbs are handled properly, we need to add another middleware, express.methodOverride, following express.bodyParser in the app.configure method call. The express.methodOverride middleware alters the HTTP method to whatever is given as value in this hidden field:

app.configure(function(){
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.static(__dirname + '/public'));
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
});

Next, we’ll need to add functionality to process these two new verbs. The update request replaces the widget object’s contents with the new contents, while the delete request deletes the widget array entry in place, deliberately leaving a null value since we do not want to reorder the array because of the widget removal.

To complete our widget application, we’ll also add in an index page for accessing widgets without any identifier or operation. All the index page does is list all the widgets currently in the memory store.

Example 7-5 shows the complete widget application with all the new functionality shown in bold text.

var express = require('express')
  , http = require('http')
  , app = express();

app.configure(function(){
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.static(__dirname + '/public'));
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
});

app.configure('development', function(){
  app.use(express.errorHandler());
});
// in memory data store
var widgets = [
  { id : 1,
    name : 'My Special Widget',
    price : 100.00,
    descr : 'A widget beyond price'
  }
]

// index for /widgets/
app.get('/widgets', function(req, res) {
   res.send(widgets);
});

// show a specific widget
app.get('/widgets/:id', function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   if (!widgets[indx])
      res.send('There is no widget with id of ' + req.params.id);
   else
      res.send(widgets[indx]);
});

// add a widget
app.post('/widgets/add', function(req, res) {
  var indx = widgets.length + 1;
  widgets[widgets.length] =
   { id : indx,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     descr : req.body.widgetdesc };
  console.log(widgets[indx-1]);
  res.send('Widget ' + req.body.widgetname + ' added with id ' + indx);
});

// delete a widget
app.del('/widgets/:id/delete', function(req,res) {
   var indx = req.params.id - 1;
   delete widgets[indx];

   console.log('deleted ' + req.params.id);
   res.send('deleted ' + req.params.id);
});

// update/edit a widget
app.put('/widgets/:id/update', function(req,res) {
  var indx = parseInt(req.params.id) - 1;
  widgets[indx] =
   { id : indx,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     descr : req.body.widgetdesc };
  console.log(widgets[indx]);
  res.send ('Updated ' + req.params.id);
});

http.createServer(app).listen(3000);

console.log("Express server listening on port 3000");

After running the application, I add a new widget, list the widgets out, update widget 1’s price, delete the widget, and then list the widgets out again. The console.log messages for this activity are:

Express server listening on port 3000
{ id: 2,
  name: 'This is my Baby',
  price: 4.55,
  descr: 'baby widget' }
POST /widgets/add 200 4ms
GET /widgets 200 2ms
GET /edit.html 304 2ms
{ id: 0,
  name: 'My Special Widget',
  price: 200,
  descr: 'A widget beyond price' }
PUT /widgets/1/update 200 2ms
GET /del.html 304 2ms
deleted 1
DELETE /widgets/1/delete 200 3ms
GET /widgets 200 2ms

Notice the HTTP PUT and DELETE verbs in bold text in the output. When I list the widgets out the second time, the values returned are:

[
  null,
  {
    "id": 2,
    "name": "This is my Baby",
    "price": 4.55,
    "descr": "baby widget"
  }
]

We now have a RESTful Express application. But we also have another problem.

If our application managed only one object, it might be OK to cram all the functionality into one file. Most applications, however, manage more than one object, and the functionality for all of those applications isn’t as simple as our little example. What we need is to convert this RESTful Express application into a RESTful MVC Express application.

If you’ve worked with most content management systems (CMS), you’ll quickly grasp the fundamentals of EJS. The following is an example of an EJS template:

<% if (names.length) { %>
  <ul>
    <% names.forEach(function(name){ %>
      <li><%= name %></li>
    <% }) %>
  </ul>
<% } %>

In the code, the EJS is embedded directly into HTML, in this example providing the data for the individual list items for an unordered list. The angle brackets and percentage sign pairs (<%, %>) are used to delimit EJS instructions: a conditional test ensures that an array has been provided, and then the JavaScript processes the array, outputting the individual array values.

The values themselves are output with the equals sign (=), which is a shortcut for “print this value here”:

<%= name %>

The value is escaped when it’s printed out. To print out an unescaped value, use a dash (-), like so:

<%- name %>

If for some reason you don’t want to use the standard open and closing EJS tags (<%, %>), you can define custom ones using the EJS object’s open and close methods:

ejs.open('<<');
ejs.close('>>');

You can then use these custom tags instead of the default ones:

<h1><<=title >></h1>

Unless you have a solid reason for doing so, though, I’d stick with the default.

Though blended with HTML, EJS is JavaScript, so you have to provide the open and closing curly braces, as well as the proper format when using the array object’s forEach method.

For the finished product, the HTML is then rendered via an EJS function call, either returning a JavaScript function that generates a result, or generating the finished result. I’ll demonstrate this once we get EJS for Node installed. Let’s do that now.

The module that’s installed is a version of EJS capable of being used with Node. It’s not the same thing you’ll get if you go to the EJS site and directly download EJS. EJS for Node can be used with client-side JavaScript, but I’m going to focus on its use with Node applications.

Install the template system using npm:

npm install ejs

Once EJS is installed, you can use it directly in a simple Node application—you don’t have to use it with a framework like Express. As a demonstration, render HTML from a given template file as follows:

<html>
<head>
<title><%= title %></title>
</head>
<body>
<% if (names.length) { %>
  <ul>
    <% names.forEach(function(name){ %>
      <li><%= name %></li>
    <% }) %>
  </ul>
<% } %>
</body>

Call the EJS object’s renderFile method directly. Doing so opens the template and uses the data provided as an option to generate the HTML.

Example 8-1 uses the standard HTTP server that comes with Node to listen for a request on port 8124. When a request is received, the application calls the EJS renderFile method, passing in the path for the template file, as well as a names array and a document title. The last parameter is a callback function that either provides an error (and a fairly readable error, at that) or the resulting generated HTML. In the example, the result is sent back via the response object if there’s no error. If there is an error, an error message is sent in the result, and the error object is output to the console.

var   http = require('http')
    , ejs = require('ejs')
;

// create http server
http.createServer(function (req, res) {


  res.writeHead(200, {'content-type': 'text/html'});

  // data to render
  var names = ['Joe', 'Mary', 'Sue', 'Mark'];
  var title = 'Testing EJS';

  // render or error
  ejs.renderFile(__dirname + '/views/test.ejs',
                 {title : 'testing', names : names},
                   function(err, result) {
      if (!err) {
         res.end(result);
      } else {
         res.end('An error occurred accessing page');
         console.log(err);
      }
  });

}).listen(8124);

console.log('Server running on 8124/');

One variation of the rendering method is render, which takes the EJS template as a string and then returns the formatted HTML:

  var str = fs.readFileSync(__dirname + '/views/test.ejs', 'utf8');

  var html = ejs.render(str, {names : names, title: title });

  res.end(html);

A third rendering method, which I won’t demonstrate, is compile, which takes an EJS template string and returns a JavaScript function that can be invoked to render HTML each time it’s called. You can also use this method to enable EJS for Node in client-side applications.

In addition to support for rendering EJS templates, EJS for Node also provides a set of predefined filters, which can further simplify the HTML generation. One filter, first, extracts out the first value in a supplied array of values. Another filter, downcase, takes the result of the first filter and lowercases the text:

  var names = ['Joe Brown', 'Mary Smith', 'Tom Thumb', 'Cinder Ella'];

  var str = '<p><%=: users | first | downcase %></p>';

  var html = ejs.render(str, {users : names });

The result is the following:

<p>joe brown</p>

The filters can be chained together, with the result of one being piped to the next. The use of the filter is triggered by the colon (:) following the equals sign (=), which is then followed by the data object. The following example of the use of filters takes a set of people objects, maps a new object consisting solely of their names, sorts the names, and then prints out a concatenated string of the names:

  var people = [
     {name : 'Joe Brown', age : 32},
     {name : 'Mary Smith', age : 54},
     {name : 'Tom Thumb', age : 21},
     {name : 'Cinder Ella', age : 16}];

  var str = "<p><%=: people | map:'name' | sort | join %></p>";
  var html = ejs.render(str, {people : people });

Here is the result of that filter combination:

Cinder Ella, Joe Brown, Mary Smith, Tom Thumb

The filters aren’t documented in the EJS for Node documentation, and you have to be careful using them interchangeably because some of the filters want a string, not an array of objects. Table 8-1 contains a list of the filters, and a brief description of what type of data they work with and what they do.

Though the application is the Widget Factory, widgets aren’t going to be the only objects the system maintains. We need to restructure the environment so that we can add objects as needed.

Currently, the environment is as follows:

/application directory
   /routes - home directory controller
   /controllers - object controllers
   /public - static files
   /views - template files

The routes and the controllers directories can stay as they are, but the views and the public directory need to be modified to allow for different objects. Instead of placing all widget views directly in views, we add them to a new subdirectory of views named, appropriately enough, widgets:

/application directory
   / views
      /widgets

Instead of placing all widget static files directly in the public directory, we also place them in a widgets subdirectory:

/application directory
   /public
      /widgets

Now, we can add new objects by adding new directories, and we’ll be able to use filenames of new.html and edit.ejs for each, without worrying about overwriting existing files.

Note that this structure assumes we may have static files for our application. The next step is to figure out how to integrate the static files into the newly dynamic environment.

The first component of the application to refine is adding a new widget. This consists of two parts: displaying a form to get the new widget’s information, and storing the new widget into the existing widget data store.

We can create an EJS template for the form, though it won’t have any dynamic components—or at least, it won’t at this time with the way the page is designed. However, it makes no sense to serve something through the template system that doesn’t need the system’s capability.

We could also just change how the form is accessed—instead of accessing the form using /widgets/new, we’d access it as /widgets/new.html. However, this introduces an inconsistency into the application routing. In addition, if we add dynamic components later to the form page, then we’ll have to change the references to the new form.

A better approach is to handle the routing request and serve the static page as if it were dynamic, but without using the template system.

The Express resource object has a redirect method we could use to redirect the request to the new.html file, but new.html is what’s going to show up in the address bar on the browser when finished. It also returns a 302 status code, which we don’t want. Instead, we’ll use the resource object’s sendfile method. This method takes as parameters a file path, possible options, and an optional callback function with an error as its only parameter. The widget controller is using only the first parameter.

The filepath is:

__dirname + "/../public/widgets/widget.html"

We used the relative indicator .. since the public directory is located off the controllers directory’s parent. However, we can’t use this path in sendfile as is, because it generates a 403 forbidden error. Instead, we use the path module’s normalize method to convert the path’s use of relative indicators into the equivalent absolute path:

// display new widget form
exports.new = function(req, res) {
    var filePath = require('path').normalize(__dirname +
                                                  "/../public/widgets/new.html");
    res.sendfile(filePath);
};

The HTML page with the form is nothing to get excited about—just a simple form, as shown in Example 8-3. However, we did add the description field back in to make the data a little more interesting.

<!doctype html>
<html lang="en">
<head>
 <meta charset="utf-8" />
 <title>Widgets</title>
</head>
<body>
<h1>Add Widget:</h1>

<form method="POST" action="/widgets/create"
enctype="application/x-www-form-urlencoded">

 <p>Widget name: <input type="text" name="widgetname"
 id="widgetname" size="25" required/></p>
 <p>Widget Price: <input type="text"
 pattern="^\$?([0-9]{1,3},([0-9]{3},)*[0-9]{3}|[0-9]+)(.[0-9][0-9])?$"
 name="widgetprice" id="widgetprice" size="25" required/></p>

 <p>Widget Description: <br />
 <textarea name="widgetdesc" id="widgetdesc" cols="20"
 rows="5"></textarea>
 <p>

 <input type="submit" name="submit" id="submit" value="Submit"/>
 <input type="reset" name="reset" id="reset" value="Reset"/>
 </p>
 </form>
</body>

The form is POSTed, of course.

Now that the application can display the new widget form, we need to modify the widget controller to process the form posting.

npm install express-rewrite

Prior to adding in the new template support, we need to make changes to the main application file to incorporate the use of the EJS template system. I won’t repeat the app.js file completely from Example 7-8 in Chapter 7, because the only change is to the configure method call to include the EJS template engine and views directory:

app.configure(function(){
  app.set('views', __dirname + '/views');
  app.set('view engine', 'ejs');
  app.use(express.favicon());
  app.use(express.logger('dev'));
  app.use(express.staticCache({maxObjects: 100, maxLength: 512}));
  app.use(express.static(__dirname + '/public'));
  app.use(express.bodyParser());
  app.use(express.methodOverride());
  app.use(app.router);
  app.use(express.directory(__dirname + '/public'));
  app.use(function(req, res, next){
    throw new Error(req.url + ' not found');
  });
  app.use(function(err, req, res, next) {
    console.log(err);
    res.send(err.message);
  });
});

Now we’re ready to convert the widget controller so it uses templates, starting with the code to add a new widget.

The actual processing of the data in the widget controller for the new widget doesn’t change. We still pull the data from the request body, and add it to the in-memory widget store. However, now that we have access to a template system, we’re going to change how we respond to the successful addition of a new widget.

I created a new EJS template, named added.ejs, shown in Example 8-4. All it does is provide a listing of the widget’s properties, and a message consisting of the title sent with the widget object.

<head>
<title><%= title %></title>
</head>
<body>
<h1><%= title %> | <%= widget.name %></h1>
<ul>
<li>ID: <%= widget.id %></li>
<li>Name: <%= widget.name %></li>
<li>Price: <%= widget.price.toFixed(2) %></li>
<li>Desc: <%= widget.desc %></li>
</ul>
</body>

The code to process the update is little different from that shown in Chapter 7, other than the fact that we’re now rendering a view rather than sending a message back to the user (the part that changes is in bold text):

// add a widget
exports.create = function(req, res) {

  // generate widget id
  var indx = widgets.length + 1;

  // add widget
  widgets[widgets.length] =
   { id : indx,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     desc : req.body.widgetdesc };

  //print out to console and confirm addition to user
  console.log(widgets[indx-1]);
  res.render('widgets/added', {title: 'Widget Added', widget : widgets[indx-1]});
};

The two options sent to the view are the page title and the widget object. Figure 8-1 shows the informative, though plain, result.

Figure 8-1. Confirmation of the added widget

The next two processes to convert to templates, update and deletion, require a way of specifying which widget to perform the action on. In addition, we also need to convert the index display of all widgets. For all three processes, we’re going to use a view to create both the widgets index page and a picklist, covered next.

A picklist is nothing more than a list of choices from which one can choose. For the widget application, a picklist could be a selection or drop-down list incorporated into an update or delete page, using Ajax and client-side scripting, but we’re going to incorporate the functionality into the widget application’s index page.

Right now, the widget application index page just has a data dump of the widget data store. It’s informative, but ugly and not all that useful. To improve the result, we’re going to add a view to display all of the widgets in a table—one widget per row, with the widget properties in columns. We’re also going to add two more columns: one with a link to edit the widget, and one with a link to delete the widget. These provide the missing piece to the application: a way to edit or delete a widget without having to remember or know its identifier.

Example 8-5 has the contents of the template for this new view, named index.ejs. Since the file is located in the widgets subdirectory, we don’t have to worry that it’s the same name as the higher-level index.ejs file.

<!doctype html>
<html lang="en">
<head>
 <meta charset="utf-8" />
 <title><%= title %></title>
</head>
<body>
<% if (widgets.length) { %>
  <table>
  <caption>Widgets</caption>
  <tr><th>ID</th><th>Name</th><th>Price</th><th>Description</th></tr>
    <% widgets.forEach(function(widget){ %>
       <tr>
       <td><%= widget.id %></td>
       <td><%= widget.name %></td>
       <td>$<%= widget.price.toFixed(2) %></td>
       <td><%= widget.desc %></td>
       <td><a href="/widgets/<%= widget.id %>/edit">Edit</a></td>
       <td><a href="/widgets/<%= widget.id %>">Delete</a></td>
       </tr>
    <% }) %>
  </table>
<% } %>
</body>

The controller code to trigger this new view is extremely simple: a call to render the view, sending the entire array of widgets through as data:

// index listing of widgets at /widgets/
exports.index = function(req, res) {
   res.render('widgets/index', {title : 'Widgets', widgets : widgets});
};

In Example 8-5, if the object has a length property (is an array), its element objects are traversed and their properties are printed out as table data, in addition to the links to edit and delete the object. Figure 8-2 shows the table after several widgets have been added to our in-memory data store.

Figure 8-2. Widget display table after the addition of several widgets

The link (route) to delete the object is actually the same as the link (route) to show it: /widgets/:id. We’ll add a mostly hidden form to the Show Widget page that includes a button to delete the widget if it’s no longer needed. This allows us to incorporate the necessary trigger for the deletion without having to add a new route. It also provides another level of protection to ensure that users know exactly which widget they’re deleting.

To display an individual widget is as simple as providing a placeholder for all of its properties, embedded into whatever HTML you want to use. In the widget application, I’m using an unordered list (ul) for the widget properties.

Since we’re also encompassing into the page the trigger for deleting the object, at the bottom I’ve added a form that displays a button reading “Delete Widget.” In the form is the hidden _method field that is used to generate the HTTP DELETE verb that routes to the application’s destroy method. The entire template is shown in Example 8-6.

<!doctype html>
<html lang="en">
<head>
 <meta charset="utf-8" />
 <title><%= title %></title>
</head>
<body>
<h1><%= widget.name %></h1>
<ul>
<li>ID: <%= widget.id %></li>
<li>Name: <%= widget.name %></li>
<li>Price: $<%= widget.price.toFixed(2) %></li>
<li>Description: <%= widget.desc %></li>
</ul>
<form method="POST" action="/widgets/<%= widget.id %>"
enctype="application/x-www-form-urlencoded">

 <input type="hidden" value="delete" name="_method" />

 <input type="submit" name="submit" id="submit" value="Delete Widget"/>
</form>

</body>

Very little modification is required in the controller code for either the show or the destroy methods. I’ve left the destroy method as is for now. All it does is delete the object from the in-memory store and send a message back to this effect:

exports.destroy = function(req, res) {
   var indx = req.params.id - 1;
   delete widgets[indx];

   console.log('deleted ' + req.params.id);
   res.send('deleted ' + req.params.id);
};

The show method required little change—simply replacing the send message with a call to render the new view:

// show a widget
exports.show = function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   if (!widgets[indx])
      res.send('There is no widget with id of ' + req.params.id);
   else
      res.render('widgets/show', {title : 'Show Widget', widget : widgets[indx]});
};

Figure 8-3 demonstrates what the Show Widget page looks like, complete with the Delete Widget button at the bottom.

Figure 8-3. The Show Widget page with delete button

By now, you’ve seen how simple it is to incorporate views into the application. The best thing of all about this system is that you can incorporate changes into the view templates without having to stop the application: it uses the changed template the next time the view is accessed.

One last view for the update widget, and we’re done converting the widget application to use the EJS template system.

The form to edit the widget is exactly the same as the form to add a widget, except for the addition of one field: _method. In addition, the form is prepopulated with the data for the widget being edited, so we need to incorporate the template tags and appropriate values.

Example 8-7 contains the contents for the edit.ejs template file. Note the use of the template tags with the value fields in the input elements. Also notice the addition of the _method field.

<!doctype html>
<html lang="en">
<head>
 <meta charset="utf-8" />
 <title><%= title %></title>
</head>
<body>
<h1>Edit <%= widget.name %></h1>

<form method="POST" action="/widgets/<%= widget.id %>"
enctype="application/x-www-form-urlencoded">

 <p>Widget name: <input type="text" name="widgetname"
 id="widgetname" size="25" value="<%=widget.name %>" required /></p>

 <p>Widget Price: <input type="text"
 pattern="^\$?([0-9]{1,3},([0-9]{3},)*[0-9]{3}|[0-9]+)(.[0-9][0-9])?$"
 name="widgetprice" id="widgetprice" size="25" value="<%= widget.price %>" required/></p>
 <p>Widget Description: <br />
 <textarea name="widgetdesc" id="widgetdesc" cols="20"
 rows="5"><%= widget.desc %></textarea>
 <p>

 <input type="hidden" value="put" name="_method" />

 <input type="submit" name="submit" id="submit" value="Submit"/>
 <input type="reset" name="reset" id="reset" value="Reset"/>
 </p>
 </form>
</body>

Figure 8-4 shows the page with a widget loaded. All you need to do is edit the field values, and then click Submit to submit the changes.

Figure 8-4. The Edit widget view

The modification to the controller code is as simple as the other modifications have been. The Edit view is accessed using res.render, and the widget object is passed as data:

// display edit form
exports.edit = function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   res.render('widgets/edit', {title : 'Edit Widget', widget : widgets[indx]});
};

The code to process the update is very close to what we had in Chapter 7, except that instead of sending a message that the object is updated, we’re using a view. We’re not creating a new view, though. Instead, we’re using the widgets/added.ejs view we used earlier. Since both just display the object’s properties and can take a title passed in as data, we can easily repurpose the view just by changing the title:

// update a widget
exports.update = function(req, res) {
  var indx = parseInt(req.params.id) - 1;
  widgets[indx] =
   { id : indx + 1,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     desc : req.body.widgetdesc}
  console.log(widgets[indx]);
  res.render('widgets/added', {title: 'Widget Edited', widget : widgets[indx]})
};

Again, the view used doesn’t impact what route (URL) is shown, so it doesn’t matter if we reuse a view. Being able to reuse a view can save us a lot of work as the application increases in difficulty.

You’ve had a chance to see pieces of the controller code throughout these sections as we convert it to use templates. Example 8-8 is an entire copy of the changed file, which you can compare to Example 7-6 in Chapter 7 to see how easily views incorporate into the code, and how much work they can save us.

var widgets = [
   { id : 1,
     name : "The Great Widget",
     price : 1000.00,
     desc: "A widget of great value"
   }
]

// index listing of widgets at /widgets/
exports.index = function(req, res) {
   res.render('widgets/index', {title : 'Widgets', widgets : widgets});
};

// display new widget form
exports.new = function(req, res) {
    var filePath = require('path').normalize(__dirname + "/../public/widgets/new.html");
    res.sendfile(filePath);
};

// add a widget
exports.create = function(req, res) {

  // generate widget id
  var indx = widgets.length + 1;

  // add widget
  widgets[widgets.length] =
   { id : indx,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     desc : req.body.widgetdesc };

  //print out to console and confirm addition to user
  console.log(widgets[indx-1]);
  res.render('widgets/added', {title: 'Widget Added', widget : widgets[indx-1]});
};

// show a widget
exports.show = function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   if (!widgets[indx])
      res.send('There is no widget with id of ' + req.params.id);
   else
      res.render('widgets/show', {title : 'Show Widget', widget : widgets[indx]});
};

// delete a widget
exports.destroy = function(req, res) {
   var indx = req.params.id - 1;
   delete widgets[indx];

   console.log('deleted ' + req.params.id);
   res.send('deleted ' + req.params.id);
};

// display edit form
exports.edit = function(req, res) {
   var indx = parseInt(req.params.id) - 1;
   res.render('widgets/edit', {title : 'Edit Widget', widget : widgets[indx]});
};

// update a widget
exports.update = function(req, res) {
  var indx = parseInt(req.params.id) - 1;
  widgets[indx] =
   { id : indx + 1,
     name : req.body.widgetname,
     price : parseFloat(req.body.widgetprice),
     desc : req.body.widgetdesc}
  console.log(widgets[indx]);
  res.render('widgets/added', {title: 'Widget Edited', widget : widgets[indx]})
};

This is the last time you’ll see the controller code for this chapter. Yet we’re about to make a profound change to the application: we’re going to replace the template system.

In the Jade template system, HTML elements are given by their name, but you don’t use any angle brackets. Nesting is indicated by indentation. So, rather than:

<html>
<head>
<title>This is the title</title>
</head>
<body>
<p>Say hi World</p>
</body>
</html>

You have:

html
   head
      title This is it
   body
      p Say Hi to the world

The contents of both the title and the paragraph elements are just included after the element name. There are no ending tags—they’re assumed—and again, indentation triggers nesting. Another example is the following, which also makes use of both class name and identifier, as well as additional nesting:

      html
   head
      title This is it
   body
      div.content
         div#title
            p nested data

This generates:

<html>
<head>
<title>This is it</title>
</head>
<body>
<div class="content">
<div id="title">
<p>nested data</p>
</div>
</div>
</body>
</html>

If you have large bodies of content, such as text for a paragraph, you can use the vertical bar, or pipe (|), to concatenate the text:

p
  | some text
  | more text
  | and even more

This becomes:

<p>some text more text and even more</p>

Another approach is to end the paragraph element with a period (.) indicating that the block contains only text and allowing us to omit the vertical bar:

p.
  some text
  more text
  and even more

If we want to include HTML as the text, we can; it ends up being treated as HTML in the generated source:

body.
  <h1>A header</h1>
  <p>A paragraph</p>

Form elements generally have attributes, and they’re incorporated in Jade in parentheses, including setting their values (if any). The attributes need only be separated by whitespace, but I list them each on a separate line to make the template more readable.

The following Jade template:

html
   head
     title This is it
   body
      form(method="POST"
           action="/widgets"
           enctype="application/x-www-form-urlencoded")
         input(type="text"
               name="widgetname"
               id="widgetname"
               size="25")
         input(type="text"
               name="widgetprice"
               id="widgetprice"
               size="25")
         input(type="submit"
               name="submit"
               id="submit"
               value="Submit")

generates the following HTML:

<html>
<head>
<title>This is it</title>
</head>
<body>
<form method="POST" action="/widgets"
 enctype="application/x-www-form-urlencoded">
<input type="text" name="widgetname" id="widgetname" size="25"/>
<input type="text" name="widgetprice" id="widgetprice" size="25"/>
<input type="submit" name="submit" id="submit" value="Submit"/>
</form>
</body>
</html>

Now we’re going to convert the widget application so it uses Jade rather than EJS. The only change that we need to make to the widget application code is in app.js, to change the template engine:

  app.set('view engine', 'jade');

No other change is necessary to the application. None. Zip.

All of the templates share the same page layout. The layout is rather plain; there are no sidebars or footers, or any use of CSS. Because of the page’s plain nature, we haven’t minded duplicating the same layout markup in each view in the prior examples. However, if we start to add more to the overall page structure—such as sidebars, a header, and a footer—then having to maintain the same layout information in each file is going to get cumbersome.

The first thing we’re going to do in our conversion, then, is create a layout template that will be used by all the other templates.

app.set('view options', {layout: false});

Example 8-9 contains the complete layout.jade file. It uses the HTML5 doctype, adds a head element with title and meta elements, adds the body element, and then references a block called content. That’s how you include blocks of content defined in other template files.

doctype 5
html(lang="en")
  head
    title #{title}
    meta(charset="utf-8")
  body
    block content

Notice the use of the pound sign and curly braces (#{}) for the title. This is how we embed data passed to the template in Jade. The use of the identifier doesn’t change from EJS, just the syntax.

To use the new layout template, we start off each of the content templates with:

extends layout

The use of extends lets the template engine know where to find the layout template for the page view, while the use of block instructs the template engine about where to place the generated content.

You don’t have to use content for the block name, and you can use more than one block. In addition, you can also include other template files if you want to break up the layout template even further. I modified layout.jade to include a header rather than the markup directly in the layout file:

doctype 5
html(lang="en")
  include header
  body
    block content

I then defined the header content in a file named header.jade, with the following:

head
  title #{title}
  meta(charset="utf-8")

There are two things to note in the new layout.jade and header.jade code.

First, the include directive is relative. If you split the views into the following subdirectory structure:

/views
   /widgets
       layout.jade
      /standard
         header.jade

you’d use the following to include the header template in the layout file:

include standard/header

The file doesn’t have to be Jade, either—it could be HTML, in which case you’ll need to use the file extension:

include standard/header.html

Second, do not use indentation in the header.jade file. The indentation comes in from the parent file and doesn’t need to be duplicated in the included template file. In fact, if you do so, your template will generate an error.

Now that we’ve defined the layout template, it’s time to convert the EJS views into Jade.

The first view to convert from EJS to Jade is the added.ejs template, which provides feedback for the successful addition of a new widget. The template file is named added.jade (the name must be the same, though the extension different, to work with the existing controller code), and it makes use of the newly defined layout.jade file, as shown in Example 8-10.

extends layout

block content
  h1 #{title} | #{widget.name}
  ul
    li id: #{widget.id}
    li Name: #{widget.name}
    li Price: $#{widget.price.toFixed()}
    li Desc: #{widget.desc}

Notice how we can still use the toFixed method to format the price output.

The block is named content, so it integrates with the expectations of the block name set in the layout.jade file. The simplified HTML for an h1 header and an unordered list is integrated with the data passed from the controller—in this case, the widget object.

Running the widget application and adding a new widget generates the same HTML as generated with the EJS: a header and a list of widget properties for the newly added widget—all without our changing any of the controller code.

tr
  td #{widget.id}
  td #{widget.name}
  td $#{widget.price.toFixed(2)}
  td #{widget.desc}
  td
    a(href='/widgets/#{widget.id}/edit') Edit
  td
    a(href='/widgets/#{widget.id}') Delete

extends layout

block content
  table
    caption Widgets
      if widgets.length
        tr
          th ID
          th Name
          th Price
          th Description
          th
          th
        each widget in widgets
          include row

extends layout

block content
  h1 Edit #{widget.name}
  form(method="POST"
       action="/widgets/#{widget.id}"
       enctype="application/x-www-form-urlencoded")
    p Widget Name:
        input(type="text"
              name="widgetname"
              id="widgetname"
              size="25"
              value="#{widget.name}"
              required)
    p Widget Price:
       input(type="text"
             name="widgetprice"
             id="widgetprice"
             size="25"
             value="#{widget.price}"
             pattern="="^\$?([0-9]{1,3},([0-9]{3},)*[0-9]{3}|[0-9]+)(.[0-9][0-9])?$"
             required)
    p Widget Description:
       br
       textarea(name="widgetdesc"
                id="widgetdesc"
                cols="20"
                rows="5") #{widget.desc}
    p
       input(type="hidden"
             name="_method"
             id="_method"
             value="put")
       input(type="submit"
             name="submit"
             id="submit"
             value="Submit")
       input(type="reset"
             name="reset"
             id="reset"
             value="reset")

ul
  li id: #{widget.id}
  li Name: #{widget.name}
  li Price: $#{widget.price.toFixed(2)}
  li Desc: #{widget.desc}

extends layout

block content
  h1 #{title} | #{widget.name}
  include widget

extends layout

block content
  h1 #{widget.name}
  include widget
  form(method="POST"
       action="/widgets/#{widget.id}"
       enctype="application/x-www-form-urlencoded")
       input(type="hidden"
             name="_method"
             id="_method"
             value="delete")
       input(type="submit"
             name="submit"
             id="submit"
             value="Delete Widget")

extends layout

block content
  table
    caption Widgets
      if widgets.length
        tr
          th ID
          th Name
          th Price
          th Description
          th
          th
        each widget in widgets
          if widget
            include row

To use the MongoDB driver, you first have to require the module:

var mongodb = require('mongodb');

To establish a connection to MongoDB, use the mongodb.Server object constructor:

var server = new mongodb.Server('localhost',:27017, {auto_reconnect: true});

All communication with the MongoDB occurs over TCP. The server constructor accepts the host and port as the first two parameters—in this case, the default localhost and port 27017. The third parameter is a set of options. In the code, the auto_reconnect option is set to true, which means the driver attempts to reestablish a connection if it’s lost. Another option is poolSize, which determines how many TCP connections are maintained in parallel.

Once you have a connection to MongoDB, you can create a database or connect to an existing one. To create a database, use the mongodb.Db object constructor:

var db = new mongdb.Db('mydb', server);

The first parameter is the database name, and the second is the MongoDB server connection. A third parameter is an object with a set of options. The default option values are sufficient for the work we’re doing in this chapter, and the MongoDB driver documentation covers the different values, so I’ll skip repeating them in this chapter.

If you’ve not worked with MongoDB in the past, you may notice that the code doesn’t have to provide username and password authentication. By default, MongoDB runs without authentication. When authentication isn’t enabled, the database has to run in a trusted environment. This means that MongoDB allows connections only from trusted hosts, typically only the localhost address.

MongoDB collections are fundamentally equivalent to relational database tables, but without anything that even closely resembles a relational database table.

When you define a MongoDB collection, you can specify if you want a collection object to actually be created at that time, or only after the first row is added. Using the MongoDB driver, the following code demonstrates the difference between the two; the first statement doesn’t create the actual collection, and the second does:

db.collection('mycollection', function(err, collection{});
db.createCollection('mycollection', function(err, collection{});

You can pass an optional second parameter to both methods, {safe : true}, which instructs the driver to issue an error if the collection does not exist when used with db.collection, and an error if the collection already exists if used with db.createCollection:

db.collection('mycollection', {safe : true}, function (err, collection{});
db.createCollection('mycollection', {safe : true}, function(err, collection{});

If you use db.createCollection on an existing collection, you’ll just get access to the collection—the driver won’t overwrite it. Both methods return a collection object in the callback function, which you can then use to add, modify, or retrieve document data.

If you want to completely drop the collection, use db.dropCollection:

db.dropCollection('mycollection', function(err, result){});

Note that all of these methods are asynchronous, and are dependent on nested callbacks if you want to process the commands sequentially. This is demonstrated more fully in the next section, where we’ll add data to a collection.

Before getting into the mechanics of adding data to a collection, and looking at fully operational examples, I want to take a moment to discuss data types. More specifically, I want to repeat what the MongoDB driver mentions about data types, because the use of JavaScript has led to some interesting negotiations between the driver and the MongoDB.

Table 10-1 shows the conversions between the data types supported by MongoDB and their JavaScript equivalents. Note that most conversions occur cleanly—no potentially unexpected effects. Some, though, do require some behind-the-scenes finagling that you should be aware of. Additionally, some data types are provided by the MongoDB Native Node.js Driver, but don’t have an equivalent value in MongoDB. The driver converts the data we provide into data the MongoDB can accept.

Once you have a reference to a collection, you can add documents to it. The data is structured as JSON, so you can create a JSON object and then insert it directly into the collection.

To demonstrate all of the code to this point, in addition to adding data to a collection, Example 10-1 creates a first collection (named Widgets) in MongoDB and then adds two documents. Since you might want to run the example a couple of times, it first removes the collection documents using the remove method. The remove method takes three optional parameters:

In the example, the application is using a safe remove, passing in null for the first parameter (as a parameter placeholder, ensuring that all documents are removed) and providing a callback function. Once the documents are removed, the application inserts two new documents, with the second insert using safe mode. The application prints to the console the result of the second insert.

The insert method also takes three parameters: the document or documents being inserted, an options parameter, and the callback function. You can insert multiple documents by enclosing them in an array. The options for insert are:

The method calls are asynchronous, which means there’s no guarantee that the first document is inserted before the second. However, it shouldn’t be a problem with the widget application—at least not in this example. Later in the chapter, we’ll look more closely at some of the challenges of working asynchronously with database applications.

var mongodb = require('mongodb');

var server = new mongodb.Server('localhost', 27017, {auto_reconnect: true});
var db = new mongodb.Db('exampleDb', server);

// open database connection
db.open(function(err, db) {
  if(!err) {

     // access or create widgets collection
     db.collection('widgets', function(err, collection) {

       // remove all widgets documents
       collection.remove(null,{safe : true}, function(err, result) {
          if (!err) {
            console.log('result of remove ' + result);

            // create two records
            var widget1 = {title : 'First Great widget',
                         desc : 'greatest widget of all',
                         price : 14.99};
            var widget2 = {title : 'Second Great widget',
                         desc : 'second greatest widget of all',
                         price : 29.99};

            collection.insert(widget1);

            collection.insert(widget2, {safe : true}, function(err, result) {
               if(err) {
                  console.log(err);
               } else {
                  console.log(result);

                  //close database
                  db.close();
               }
            });
          }
       });
     });
  }
});

The output to the console after the second insert is a variation on:

[  { title: 'Second Great widget',
    desc: 'second greatest widget of all',
    price: 29.99,
    _id: 4fc108e2f6b7a3e252000002 } ]

The MongoDB generates a unique system identifier for each document. You can access documents with this identifier at a future time, but you’re better off adding a more meaningful identifier—one that can be determined easily by context of use—for each document.

As mentioned earlier, we can insert multiple documents at the same time by providing an array of documents rather than a single document. The following code demonstrates how both widget records can be inserted in the same command. The code also incorporates an application identifier with the id field:

             // create two records
            var widget1 = {id: 1, title : 'First Great widget',
                         desc : 'greatest widget of all',
                         price : 14.99};
            var widget2 = {id: 2, title : 'Second Great widget',
                         desc : 'second greatest widget of all',
                         price : 29.99};

            collection.insert([widget1,widget2], {safe : true},
                                                         function(err, result) {
               if(err) {
                  console.log(err);
               } else {
                  console.log(result);

                  // close database
                  db.close();
               }
            });

If you do batch your document inserts, you’ll need to set the keepGoing option to true to be able to keep inserting documents even if one of the insertions fails. By default, the application stops if an insert fails.

There are four methods of finding data in the MongoDB Native Node.js Driver:

In this section, I’ll demonstrate collection.find and collection.findOne, and reserve the other two methods for the next section, Using Updates, Upserts, and Find and Remove.

Both collection.find and collection.findOne support three arguments: the query, options, and a callback. The options object and the callback are optional, and the list of possible options to use with both methods is quite extensive. Table 10-2 shows all the options, their default setting, and a description of each.