Generators and the associated Iteration Protocol are a large topic, which we will briefly cover. 

The Iteration Protocol is what's behind the new for..of loop, and some other new looping constructs. These constructs can be used with anything producing an iterator. For more about both, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Iteration_protocols.

A generator is a kind of function which can be stopped and started using the yield keyword. Generators produce an iterator whose values are whatever is given to the yield statement. For more on this, see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator.

Consider this:

$ cat gen.js
function* gen() {
    yield 1;
    yield 2;
    yield 3;
    yield 4;
}
for (let g of gen()) {
   console.log(g);
}
$ node gen.js
1
2
3
4

The yield statement causes a generator function to pause and to provide the value given to the next call on its next function. The next function isn't explicitly seen here, but is what controls the loop, and is part of the iteration protocol. Instead of the loop, try calling gen().next() several times:

var geniter = gen();
console.log(geniter.next());
console.log(geniter.next());
console.log(geniter.next());

You'll see this:

$ node gen.js
{ value: 1, done: false }
{ value: 2, done: false }
{ value: 3, done: false }

The Iteration protocol says the iterator is finished when done is true. In this case, we didn't call it enough to trigger the end state of the iterator.

Where generators became interesting is when used with functions that return a Promise. The Promise is what's made available through the iterator. The code consuming the iterator can wait on the Promise to get its value. A series of asynchronous operations could be inside the generator and invoked in an iterable fashion. 

With the help of an extra function, a generator function along with Promise-returning asynchronous functions can be a very nice way to write asynchronous code. We saw an example of this in Chapter 2, Setting up Node.js, while exploring Babel. Babel has a plugin to rewrite async functions into a generator along with a helper function, and we took a look at the transpiled code and the helper function. The co library (https://www.npmjs.com/package/co) is a popular helper function for implementing asynchronous coding in generators. Create a file named 2files.js:

const fs = require('fs-extra');
const co = require('co');
const util = require('util');
co(function* () {
  var texts = [
    yield fs.readFile('hello.txt', 'utf8'),
    yield fs.readFile('goodbye.txt', 'utf8')
  ];
  console.log(util.inspect(texts));
});

Then run it like so:

$ node 2files.js
[ 'Hello, world!\n', 'Goodbye, world!\n' ]

Normally, fs.readFile sends its result to a callback function, and we'd build a little pyramid-shaped piece of code to perform this task. The fs-extra module contains implementations of all functions from the built-in fs module but changed to return a Promise instead of a callback function. Therefore, each fs.readFile shown here is returning a Promise that's resolved when the file content is fully read into memory. What co does is it manages the dance of waiting for the Promise to be resolved (or rejected), and returns the value of the Promise. Therefore, with two suitable text files we have the result shown from executing 2files.js.

The important thing is that the code is very clean and readable. We aren't caught up in boilerplate code required to manage asynchronous operations. The intent of the programmer is pretty clear.

async functions take that same combination of generators and Promises and define a standardized syntax in the JavaScript language. Create a file named 2files-async.js:

const fs    = require('fs-extra');
const util  = require('util');
async function twofiles() {
    var texts = [
        await fs.readFile('hello.txt', 'utf8'),
        await fs.readFile('goodbye.txt', 'utf8')
    ];
    console.log(util.inspect(texts));
}
twofiles().catch(err => { console.error(err); });

Then run it like so:

$ node 2files-async.js 
[ 'Hello, world!\n', 'Goodbye, world!\n' ]

Clean. Readable. The intent of the programmer is clear. No dependency on an add-on library, with syntax built-in to the JavaScript language. Most importantly, everything is handled in a natural way. Errors are indicated naturally by throwing exceptions. The results of an asynchronous operation naturally appear as the result of the operation, with the await keyword facilitating the delivery of that result.

To see the real advantage, let's return to the Pyramid of Doom example from earlier:

router.get('/path/to/something', async (req, res, next) => { 
   try {
       let data1 = await doSomething(req.query.arg1, req.query.arg2);
       let data2 = await doAnotherThing(req.query.arg3, req.query.arg2, 
       data1);
       let data3 = await somethingCompletelyDifferent(req.query.arg1,
                                                      req.query.arg42);
       let data4 = await doSomethingElse();
       res.render('page', { data1, data2, data3, data4 }); 
   } catch(err) {
       next(err);
   }
}); 

Other than the try/catch, this example became very clean compared to its form as a callback pyramid. All the asynchronous callback boilerplate is erased, and the intent of the programmer shines clearly.

Why was the try/catch needed? Normally, an async function catches thrown errors, automatically reporting them correctly. But since this example is within an Express router function, we're limited by its capabilities. Express doesn't know how to recognize an async function, and therefore it does not know to look for the thrown errors. Instead, we're required to catch them and call next(err).

This improvement is only for code executing inside an async function. Code outside an async function still requires callbacks or Promises for asynchronous coding. Further, the return value of an async function is a Promise.

Refer to the official specification of async functions at https://tc39.github.io/ecmascript-asyncawait/ for details.