Callback Hell

Imagine you have a function like the following:

Listing 10-1. Synchronous functions

let sync = () => {
        //some operation
        //return data
}

let sync2 = () => {
        //some operation
        //return data
}

let sync3 = () => {
        //some operation
        //return data
}

The above functions sync, sync1, and sync2 do some operations synchronously and return the results. As a result, one can call these functions like this:

result = sync()
result2 = sync2()
result3 = sync3()

What if the operation is asynchronous? Let’s see it in action:

Listing 10-2. Asynchronous functions

let async = (fn) => {
        //some async operation
        //call the callback with async operation
        fn(/*  result data */)
}

let async2 = (fn) => {
        //some async operation
        //call the callback with async operation
        fn(/*  result data */)
}

let async3 = (fn) => {
        //some async operation
        //call the callback with async operation
        fn(/*  result data */)
}

Synchronous vs. Asynchronous

Synchronous is when the function blocks the caller when it is executing and returns the result once it’s available.

Asynchronous is when the function doesn't block the caller when it’s executing the function but returns the result once available.

We deal with Asynchronous heavily when we deal with an AJAX request in our project.

Now if someone wants to process these functions at once, how they do it? The only way to do it is like this:

Listing 10-3. Async functions calling example

async(function(x){
    async2(function(y){
        async3(function(z){
            ...
        });
    });
});

Oops! You can see in the above code (Listing 10-3), we are passing many callback functions to our async functions! This little piece of code showcases what Callback Hell is! Callback Hell makes the program harder to understand. Handling errors and bubbling the errors out of callback are tricky and always error prone.

Before ES6 arrived, JavaScript developers used Promises to solve the above problem. Promises are great, but given the fact that ES6 has generators at language level, we don’t need Promises anymore!

Creating Generators

Let’s start our journey by seeing how to create generators in the first place. Generators are nothing but a function that comes up with its own syntax. A simple generator looks like the following:

Listing 10-4. First Simple Generator

function* gen() {
    return 'first generator';
}

The function gen in the above code is a generator. As you might notice, we have used an asterisk before our function name (in this case gen) to denote that it’s a generator function! Al right, we have seen how to create a generator; now let’s see how to invoke a generator:

let generatorResult = gen()

What will be the result of generatorResult? Is it going to be first generator value? Let’s print it on the console and inspect it:

console.log(generatorResult)

The result will be:

gen {[[GeneratorStatus]]: "suspended", [[GeneratorReceiver]]: Window}

The above result shows that the generatorResult is not a normal function, but an instance of Generator primitive type! So the question is how to get the value from this generator instance? The answer is to call the function next, which is available on the generator instance. So to get the value you need to do this:

generator.next()

The above code returns:

Object {value: "hello world", done: true}

As you can see the returned object from next has value and is done. So we need to call next along with fetching a value from the object:

generator.next().value
=> 'first generator'

That’s great!

Caveats of Generators

The above examples show how to create a generator, how to create an instance for it, and how it gets value. But there are a few important things we need to take care of while we are working with generators.

The first thing is that we can't call next as many times as we want to get the value from the generator. To make it clearer, let’s try to fetch a value from our first generator (Refer Listing 10-4 for first generator definition):

let generatorResult = gen()

//for the first time
generatorResult.next().value
=> 'first generator'

//for the second time
generatorResult.next().value
=> undefined

As you can see in the above code, calling next for the second time will return an undefined rather than first generator. The reason is that generators are like sequences: once the values of the sequence are consumed, you can't consume it again. In our case generatorResult is a sequence that has value as a first generator. With our first call to next, we (as the caller of the generator) have consumed the value from the sequence. Since the sequence is empty by now, calling it a second time will return you undefined!

In order to consume the sequence again, you need to create another generator instance:

let generatorResult = gen()
let generatorResult2 = gen()

//first sequence
generatorResult.next().value
=> 'first generator'

//second sequence
generatorResult2.next().value
=> 'first generator'

The above code also shows that different instances of Generators can be in different states. The key takeaway here is that each generator's state depends on how we are calling the next function on it.

yield New Keyword

With generator functions, there is a new keyword that we can use called yield. In this section, we are going to see how to use yield within a generator function.

Let’s start with the below code:

Listing 10-5. Simple Generator Sequence

function* generatorSequence() {
    yield 'first';
    yield 'second';
    yield 'third';
}

As usual we can create a generator instance for the above code:

let generatorSequence = generatorSequence();

Now if we call next for the first time we get back the value first:

generatorSequence.next().value
=> first

What happens if we call next again? Do we get first? Or second? Or third? Or Error? Let’s find that out:

generatorSequence.next().value
=> second

What? We got back the value second, how come? yield makes the generator function pause the execution and send back the result to the caller. So when we call generatorSequence for the first time, the function sees the yield with value first, so it put the function to pause mode and returned back the value (and it remembers where it exactly paused, too). The next time, we call the generatorSequence (using the same instance variable), the generator function resumes from where it left off. Since it paused at the line:

yield 'first';

for the first time, when we call it for second time (using the same instance variable), we get back the value second. What happens when we call it for the third time? Yeah, we will get back the value third!

This is better explained by looking at Figure 10-1:

Figure 10-1. Visual View Of Generator Listed In 10-4

This sequence is explained via the code here:

Listing 10-6. Calling our generator sequence

//get generator instance variable
let generatorSequenceResult = generatorSequence();

console.log('First time sequence value',generatorSequenceResult.next().value)
console.log('Second time sequence value',generatorSequenceResult.next().value)
console.log('thrid time sequence value',generatorSequenceResult.next().value)

prints back to the console:

First time sequence value first
Second time sequence value second
third time sequence value third

With that understanding in place, you can see why we call a generator as a sequence of values! One more important point to keep in mind is that all generators with yield will execute in lazy evaluation order.

Lazy evaluation

What is lazy evaluation? To put it in simple terms, lazy evaluation means the code won’t run until we ask it to run. As you can guess, the example of generatorSequence function shows that Generators are lazy evaluated. The values are being executed and returned only when we ask for them. That’s so lazy about Generators, isn't it?

done Property of Generator

Now we have seen how a generator can produce a sequence of values lazily with the yield keyword. But a generator can produce n numbers of sequence; as a user of the generator function, how will you know when to stop calling next? Because calling next on your already consumed generator sequence will give back the undefined value. How to handle this situation? This is where the done property comes into the picture.

Remember that every call to the next function is going to return back an object that looks like the following:

{value: 'value', done: false}

We are aware that the value is the value from our generator. But what about done? done is a property that is going to tell whether the generator sequence has been fully consumed or not.

We will rerun the code from previous sections here (Listing 10-4), just to print the object being returned from the next call:

Listing 10-7. Code For Understanding done property

//get generator instance variable
let generatorSequenceResult = generatorSequence();

console.log('done value for the first time',generatorSequenceResult.next())
console.log('done value for the second time',generatorSequenceResult.next())
console.log('done value for the third time',generatorSequenceResult.next())

Running the above code will print the following:

done value for the first time { value: 'first', done: false }
done value for the second time { value: 'second', done: false }
done value for the third time { value: 'third', done: false }

As you can see we have consumed all the values from the generator sequence, so calling next again will return the following object:

console.log(generatorSequenceResult.next())
=> { value: undefined, done: true }

Now the done property clearly tells us that the generator sequence is already fully consumed! When the done is true, it’s time for us to stop calling next on that particular generator instance! Again it can be better visualized with Figure 10-2:

Figure 10-2. Visual View Of Generators done property for generatorSequence

Since Generator become the core part of ES6, we have a for loop that will allow us to iterate a generator (after all it’s a sequence. :)

for(let value of generatorSequence())
        console.log("for of value of generatorSequence is",value)

which is going to print:

for of value of generatorSequence is first
for of value of generatorSequence is second
for of value of generatorSequence is third

notably for using the generator’s done property to iterate through it!

Passing Data to Generators

In this section, let’s discuss how we pass data to generators. Passing data to generators might feel confusing at first, but as you will see in this chapter, it makes async programming easy!

Let’s take a look at the following code snippet:

Listing 10-8. Passing Data Generator Example

function* sayFullName() {
    var firstName = yield;
    var secondName = yield;
    console.log(firstName + secondName);
}

The code snippet now might be not a surprise for you. Let’s use this code to explain the concept of passing data to the generator. As always, we create a generator instance first:

let fullName = sayFullName()

Once the generator instance is created, let’s call next on it:

fullName.next()
fullName.next('anto')
fullName.next('aravinth')
=> anto aravinth

In the above code snippet the last call will print anto aravinth to the console! You might be confused with this result, so let’s walk over the code slowly. When we call the next for the first time:

fullName.next()

the code will return and pause at the line:

var firstName = yield;

Since here we are not sending any value back via yield, next will return a value undefined. The second call to next is where an interesting thing happens:

fullName.next('anto')

Here we are passing value anto to the next call! Now the generator will be resumed from its previous paused state, if you remember the previous paused state is on the line:

var firstName = yield;

Since we have passed value anto on this call, yield will be replaced by anto and thus firstName holds the value anto. After the value is being set to firstName, the execution will be resumed (from the previous paused state) and again sees the yield and stops the execution at:

var secondName = yield;

Now for the third time, if we call next:

fullName.next('aravinth')

When this line gets executed, our generator will resume from where it paused. The previous paused state is:

var secondName = yield;

As before, the passed value aravinth of our next call will be replaced by yield and aravinth is set to secondName. Then the generator happily resumes the execution and sees the statement:

console.log(firstName + secondName);

By now, firstName is anto and secondName is aravinth, so the console will print anto aravinth.

This full process is explained in Figure 10-3:

Figure 10-3. Explaining How Data Is Passed To sayFullName Generator

You might be wondering why we need such an approach. It turns out that using generators by passing data to them makes it very powerful. We will be using the same technique in the next section to handle async calls!

Generators for Async - A Simple Case

In this section, we are going to see how to use generators for handling async code. Since we are getting started with a different mindset of using generators to solve the async problem, I want it to keep the section simple. So we will be mimicking the async calls with setTimeout calls!

Imagine you have two functions (which are async in nature):

Listing 10-9. Simple Asynchronous Functions

let getDataOne = (cb) => {
        setTimeout(function(){
        //calling the callback
        cb('dummy data one')
    }, 1000);
}

let getDataTwo = (cb) => {
        setTimeout(function(){
        //calling the callback
        cb('dummy data two')
    }, 1000);
}

Both of the above functions mimics the async code with setTimeout. Once the desired time has elapsed, setTimeout will call the passed callback cb with value dummy data one and dummy data two respectively. Let’s see how we will be calling these two functions without generators in the first place:

getDataOne((data) => console.log("data received",data))
getDataTwo((data) => console.log("data received",data))

The above code will print:

data received dummy data one
data received dummy data two

after 1000ms.

Now as you notice, we are passing the callbacks to get back the response. We have talked about how bad the Callback Hell can be in async code. Let’s use our generator knowledge to solve the current problem. We now change both the functions getDataOne and getDataTwo to use generator instances rather than callbacks for passing the data.

First let’s change the function getDataOne(Listing 10-8) to:

Listing 10-10. Changing getDataOne to use generator

let generator;
let getDataOne = () => {
        setTimeout(function(){
        //call the generator and
        //pass data via next
        generator.next('dummy data one')
    }, 1000);
}

We have changed the callback line from:

. . .
cb('dummy data one')
. . .

to

generator.next('dummy data one')

That’s a simple change. Note that we have also removed the cb, which is not required in this case. We will do the same for getDataTwo (Listing 10-8) too:

Listing 10-11. Changing getDataTwo to use generator

let getDataTwo = () => {
        setTimeout(function(){
        //call the generator and
        //pass data via next
        generator.next('dummy data two')
    }, 1000);
}

Now with that change in place, let’s go and test our new code. We’ll wrap our call to getDataOne and getDataTwo inside a separate generator function:

Listing 10-12. main generator function

function* main() {
    let dataOne = yield getDataOne();
    let dataTwo = yield getDataTwo();
    console.log("data one",dataOne)
    console.log("data two",dataTwo)
}

Now the main code looks exactly like sayFullName function from our previous section. Let’s create a generator instance for main and trigger the next call and see what happens:

generator = main()
generator.next();

which will print the following to the console:

data one dummy data one
data two dummy data two

which is what exactly we wanted. Look at our main code; the code looks like synchronous calls to the function getDataOne and getDataTwo. However both these calls are asynchronous. Remember that these calls never block and they work in async fashion! Let’s distill how this whole process works.

First we are creating a generator instance for main using the generator variable that we declared earlier. Remember that this generator is used by both getDataOne and getDataTwo to push the data to its call, which we will see soon. After creating the instance, we are firing the whole process with the line:

generator.next()

This calls the main function. main function is put into execution and we see the first line with yield:

. . .
let dataOne = yield getDataOne();
. . .

Now the generator will be put into pause mode as it has seen a yield statement. But before it’s been put into pause mode, it calls the function getDataOne.

An important point to note here is that even though the yield makes the statement pause, it won’t make the caller wait (i.e., caller is not blocked). To make the point more concrete, let’s see the below code:

generator.next() //even though the generator pause for Async codes

console.log("will be printed")
=> will be printed
=> Generator data result is printed

The above code shows that even though our generator.next causes the Generator function to wait on the next call, the caller (the one who is calling the generator) won’t be blocked! As you can see above, console.log will be printed (showcasing generator.next isn't blocked), and then we get the data from the generator once the async operation is done!

Now interestingly getDataOne function has the following line in its body:

. . .
    generator.next('dummy data one')
. . .

As we discussed in a previous section, calling next by passing parameter will resume the paused yield! And that’s exactly what happens here in this case too! Remember that this piece of line is inside setTimeout. So this line will get executed only when 1000ms have elapsed. Still then, the code will be paused at the line:

let dataOne = yield getDataOne();

One more important point to note here is that while this line is paused, the timeout will be running down from 1000 to 0. Once it reaches 0, it is going to execute the line:

. . .
    generator.next('dummy data one')
. . .

which is going to send back dummy data one to our yield statement. So the dataOne variable becomes dummy data one:

//after 1000ms dataOne becomes
//`dummy data one`
let dataOne = yield getDataOne();
=> dataOne = `dummy data one`

That’s a lot of interesting stuff going around! And once dataOne is set to dummy data one value, the execution will continue to the next line:

. . .
let dataTwo = yield getDataTwo();
. . .

This line is going to run the same way as the line before! So after the execution of this line, we have dataOne and dataTwo:

dataOne = dummy data one
dataTwo = dummy data two

which is what is getting printed to the console at the final statements of the main function:

. . .
    console.log("data one",dataOne)
    console.log("data two",dataTwo)
. . .

The full process is shown in Figure 10-4:

Figure 10-4. Image Explainging How main generators works internally

Phew! Now you have made an Asynchronous call look like a Synchronous call, but it works in an Asynchronous way!

Generators for Async - A Real-World Case

In the previous section, we saw how to handle asynchronous code using generators effectively. To mimic the async workflow we have used setTimeout. In this section, we are going to use a function to fire a real AJAX call to Reddit APIs to showcase the power of generators in the real world!

To make a async call, let’s create a function called httpGetAsync:

Listing 10-13. httpGetAsync function definition

let https = require('https');
function httpGetAsync(url,callback) {

    return https.get(url,
        function(response) {
            var body = '';
            response.on('data', function(d) {
                body += d;
            });
            response.on('end', function() {
                let parsed = JSON.parse(body)
                callback(parsed)
            })
        }
    );

}

This is a simple function that uses a https module from node to fire an AJAX call to get the response back.

Here we are not going to see in detail how httpGetAsync function works. The problem we are trying to solve is how to convert functions like httpGetAsync, which works the async way but expects a callback to get the response from AJAX calls.

Let’s check the httpGetAsync by passing a reddit URL:

httpGetAsync('https://www.reddit.com/r/pics/.json',(data)=> {
        console.log(data)
})

It works by printing the data to the console. The URL https://www.reddit.com/r/pics/.json prints the list of json about Picture Reddit page. The returned json has a data key whose structure looks like the following:

{ modhash: '',
  children:
   [ { kind: 't3', data: [Object] },
     { kind: 't3', data: [Object] },
     { kind: 't3', data: [Object] },
     . . .
     { kind: 't3', data: [Object] } ],
  after: 't3_5bzyli',
  before: null }

Imagine we want to get the URL of the first children of the array; we need to navigate to data.children[0].data.url. This will give us a URL like https://www.reddit.com/r/pics/comments/5bqai9/introducing_new_rpics_title_guidelines/ . Since we need to get the json format of the given URL, we need to append .json to the URL, so that it becomes https://www.reddit.com/r/pics/comments/5bqai9/introducing_new_rpics_title_guidelines/.json .

Now let’s see that in action:

httpGetAsync('https://www.reddit.com/r/pics/.json',(picJson)=> {
    httpGetAsync(picJson.data.children[0].data.url+".json",(firstPicRedditData) => {
        console.log(firstPicRedditData)
    })
})

The above code will print the data as required. We are least worried about the data being printed. But we are worried about our code structure. As we have seen in the beginning of the chapter, code that looks like this suffers from Callback Hell. Here there are two levels of callbacks that might not be a real problem. But what if it goes to 4-5 nested levels? Can you read such codes easily? Definitely not. Now let’s find out how to solve the problem via generator!

Let’s wrap httpGetAsyncinside a separate method called request:

Listing 10-14. request function

function request(url) {
    httpGetAsync( url, function(response){
        generator.next( response );
    } );
}

We have removed the callback with the generators’ next call, very similar to our previous section. Now let’s wrap our requirement inside a generator function; again we call it main:

Listing 10-15. main generator function

function *main() {
    let picturesJson = yield request( "https://www.reddit.com/r/pics/.json" );
    let firstPictureData = yield request(picturesJson.data.children[0].data.url+".json")
    console.log(firstPictureData)
}

The above main function looks very similar to the main function that we have defined in Listing 10-11 (only change is method call details). In the code we are yielding on two calls to request. As you have seen in the setTimeout example, calling yield on request will make it pause until request calls the generator next by sending the AJAX response back! The first yield will get the json of pictures, and the second yield gets the first picture data by calling request respectively! Now we have made the code look like synchronous code, but in reality it works in asynchronous fashion!

We have also escaped from Callback Hell using generators. Now the code looks clean and clearly tells what it’s doing! That’s so much more power for us!

Try running it:

generator = main()
generator.next()

It’s going to print the data as required! We have clearly seen how to use generators to convert any function that expects a callback mechanism into a generator based one. In turn, we get back clean code for handling an asynchronous operation.