The Filesystem Module

Suppose you want to open a file and read data. Node.js has no functions for files built-in, but it does have a nice module to work with files.

To load the filesystem module, you require its reference fs. This generally looks as follows:

var fs = require('fs');

By loading the fs module, you have access to many objects and classes that help you to work with files in JavaScript.

Let’s play with the module in the Node.js console. First, you can open a Node.js console with:

$ node

Then, you require the module:

> var fs = require('fs');
{ Stats: [Function],
  ...

As you can see, the JavaScript object fs has many functions for working with files.

Within the object’s methods, you will find two interesting functions to read a file, both starting with readFile:

fs.readFile
fs.readFileSync

The filesystem module provides both “synchronous” functions and “nonblocking” or “asynchronous” functions. The “sync” version of a function blocks code execution until the file is completely read. This can be useful for reading smaller files where you don’t expect a large delay from reading. However, blocking the JavaScript event loop is not nice for other functions. When working with smaller files, reading a file synchronously can be OK as long as you don’t expect a large delay from reading.

Besides loading a module with require, you can share modules by using the module.exports syntax across different projects.

Writing a module yourself is very simple too. You basically declare what part of code you want to make available to others with:

var myFunction = function() {
  console.log('my function');
}
module.exports = myFunction;

You could require this module similarly to requiring the filesystem module. Say you have some code to parse a CSV file. A simplified script might look as follows in a file called parseCSV.js:

var fs = require('fs');

// read a CSV file and return its values
var parseCSV = function(filename) {
  var data = fs.readFileSync(filename);
  return data.toString().split(',');
}

Again, the first thing you see in the script is the reference to the filesystem module. Then comes the declaration of a JavaScript function parseCSV. Inside this function, we read a file with the synchronous version (when working with small files, this is often the easier approach). Finally, the values from the CSV file are returned.

To read a file from the command line, you add these lines:

var fields = parseCSV(process.argv[2]);
console.log(fields);

To test this code, you can run from the command line:

$ node parseCSV.js test.csv
[ '10:12', 'greeting', 'hello_world', '1', '88\n' ]

Now, to turn this parser into a Node.js module, you can make a module parseCSV that you can require in other places. Therefore, you must explicitly declare what function to export:

module.exports = parseCSV;

The resulting module parseCSV.js file is:

var fs = require('fs');

// read a CSV file and return its values
var parseCSV = function(filename) {
  var data = fs.readFileSync(filename);
  return data.toString().split(',');
}
module.exports = parseCSV;

The require and module.exports syntax are the basics of working with Node.js modules.

Install Modules with npm

Working with npm has two aspects: you declare the project’s dependencies in a manifest file package.json (which can easily be installed with npm), and npm provides some commands to run specific scripts for your project.

Let’s start by looking at how to install modules.

A Node project defines its module dependencies in a file called package.json. For new projects, you can set up a project manifest with:

$ npm init

Inside of the package.json file, you’ll find a number of default settings:

{
  "name": "sandbox",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
  }
}

To install a module to find different files, you can use the findit module by @substack:

$ npm install --save serialport

The module will be installed in a local folder /node_modules, and you will see the dependency including its version listed in the package.json file.

Running Scripts with npm

In the project manifest package.json, there is the possibility to save commands for running scripts. Under a field scripts, you can specify a command to execute from the command line. For example, you might want to add a command to check your network connectivity. To do this, you would extend the scripts sections with:

"scripts": {
   "ping": "ping npmjs.com"
 }

The server npmjs.com is the address of the npm. You’ll find plenty of useful modules there, as you will see in a second. Let’s test the new script:

$ npm run ping

> serial-test@1.0.0 ping /Users/pmu/projects/node/sandbox
> ping npmjs.com

PING npmjs.com (54.187.170.181): 56 data bytes
64 bytes from 54.187.170.181: icmp_seq=0 ttl=47 time=189.291 ms
64 bytes from 54.187.170.181: icmp_seq=1 ttl=47 time=197.831 ms

This looks like a response from a network. You could also test connections to the serial port, list information about serial ports, or upload a default Arduino sketch.

The Stream Module

If you receive data over a network or the serial port, you might not know how much and when data becomes available. Instead of blocking code execution or managing execution threads, Node.js provides streams to solve this problem. With streams, you can observe and manipulate the flow of data over time. A stream is a bit like a river where data “flows.”

Streams provide a mechanism to work with chunks of data, whenever data is available.

The simplest stream is a readable stream that emits events when data is read or received. As a first step, let’s write a stream that echoes the received data:

var fs = require('fs');

// create a stream from a file
var rs = fs.createReadStream('test.csv');

rs.on('data', function(buffer) {
   console.log('stream: ' + buffer.toString());
});

What happens is this: we create a data source from reading a file. This stream “rs” will emit a “data” event after reading a line. The received data is a buffer type. We can convert a buffer to a string with toString().

A writable stream executes a write function when the Node.js VM allows it to. We add this function to the stream:

ws.write = function(data) {
  console.log("input=" + data);
}

ws.end = function(data) {
  console.log("bye");
}

Finally, we pipe the input from the command line into the stream with:

process.stdin.pipe(ws);

A simple test shows how this works:

$ node pipe_out.js
hello
input=hello

You could also pipe the output of a file into the write stream as follows:

$ echo hello | node pipe_out.js
input=hello

bye

The command line, requests from networks, and data from the serial port will all be based on streams.